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ABSTRACT 

The 'first half of this report, concerned with the 
measurement of student achievement in reading comprehension, , 
Following the introduction there, is a brief qritique of standardisjed, 
norm-referenced tests o'f stude'nt performance in reading. It i!s\Sirgued 
that school districts need tests that' are sensitiv.e to instructional 
treatments rather than more global measures of reading ability. In 
the section. New Measures of Student Achievement, comprehe^nsion is 
defined and then a multiple-choice cloze testing system is presented. 
In the next section of the paper the ' measurement of prqi^Aam resources 
is discussed.' The measurement of resources utilization in units of 
,time is offered as the most accurate,, useful assessment of program 
iresources. Finally, the work on the measurement of student 
achievement and program resources >and aetlioclology in productivity 
analyses is summarized, and a statewide survey of productivity in 
reading programs is proposed. The Technical . Appendix provides a 
detailed summary of procedures, r.esults, and^survey instruments. It 
is written for the research specialist interested in the Main 
Heport's conclusions. It is designed to clarify procedures \and 
results and to present studies which will help" reduce disagreement 
Over some of the researcli approaches employed. (EC) 
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Preface ^ . 

American concern with equality of educational opportunity in the 1950' s 
ai^ 60* s and the inception of massive Federal support for education witK the 
Elementary and Secondary Education Act of 1965 brought unprecedented attention 
to educational evaluation. Federal funding required the follow-up of special 
programs and added ''accountability^' to the vocabulary of education. School 
programs, however, failed to produce gains in student achievement as measured 
by traditional tests, and the demands for accountability, fanned by the cur- 
rent fiscal climate, have intensified* Today, educational systerps must dem- 
onstrate that students are achievilpig or that pTrograms can be modified to in- ; 
crease achievement irrespective of budgetary strictures. 

Many educational researchers have become increasingly critical of the 
estimates of student achievement and resource utilization used in program 
evaluations. Legitimate productivity analyses of school programs cannot be 
implemented without accurate, comparable measures of student achievement on 
the one hand and the actual use of program resources on the other. 

As a consequence of these considerations, the Bureau of School and 
Cultural Research of the State Education Department has directed its research 
and development efforts toward more appropriate and sensitive testing instruments 
and the adaptation of productivity analyses to school programs. Specifically, 
the bureau has developed an accurate, sensitive, and economical system for 
measuring, literal comprehension, a ba^ic skill in reading programs, and has 
conducted a pilot study of the feasibility of productivity analyses read- 
Ing programs. Both activities are described In this report, in addition, a 
statewide survey of productivity in reading programs is proposed. 

Organization and Brief Summary of the Report 

if , 
The first half of this report is concerned with the measuremen,t of student 



ERLC 



a ■ . 

-iii- 



achievement in reading comprehension. Following the Introduction, which 
places the measurement of student achievement and program resources in his- 
.torical perspective, there is a brief critique of standardized, norm-refer- 
enced tests of student perfomance in reading. The bureau argues that school 
districts need reading tests that are sensitive to instructional treatments 
rather than more global measures of reading ability. The hallmarks of an 
appropriate test of student achievement in reading comprehension are disr- 
cusW. In the section. New Measures of Student Achievement, comprehension 
per se is first defined, and then a multiple-choice cloze testing system is 
presented as an accurate, appropriate, and economical means of satisfying 
several evaluative needs of school districts. 

In the next section of the paper. Empirical Investigations to Date, the 
measurement of program resources is discussed. The measurement of resource 
utilization in units of time is offered as the most accurate, useful assess- 
ment of program resources. In a pilot study of the feasibility of produc- 
tivity analyses of reading programs conducted in the spring of 1974, the 
bureau demonstrated" that it is possible to gather large quantities of data 
about the allocation of instructional time in reading programs. In addition, , 
the study indicated clearly that criterion-referenced tests ate morfe sensi- 
tive to the allocation of instructional time than traditional, norm-referenced 
tests. 

Finally, in, the Conclusions jj^^<i!^o j ected Research, the work of the^ 
bureau on the measurement of student achievement and program resources and 
methodology in productivity analyses is summarized, and a statewide survey 
of productivity in reading programs is proposed. 
The Technical Appendix 

4 t 

' The bureau has developed a detailed Technical Appendix to supplement 
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the main report- The Technical Appendix is avaiba^le on request from the 
Bureau of School and Cultural llesearch. * ^ 
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Introduction 

' The belie£^hat education directly and posi5^velj^f £ects the quality 
of life of the citizenry, and thereby the quality and vitality of the nation 
as a whole, is deeply rooted in the American consciousness.. It is at least 
as^old as Thomas Jefferson's "Letter to John Adams"^ where the relationship . 
between education and the state is clearly formulated.^ Believing that a 
free state demanded an enlightened body politic, Jefferson recommended public 
funding for 3 years of schooling for all free citizens and higher edu- 
catiDn for those students with^^intellectual promi5e. To Jefferson, public 
education at the elemenHary level meant skills acquisition— reading, writing,, 
and common arithmetic— basic skills- thatf would allow the citizenry to gain 
information and,' hoifefully, to use' it rationally in the improvement of tiie - 
common weal." Beyond that simple triumvirate of educational skills, Jefferson 
clearly implied another aspect of education, socialization, when he stressed 
the use of both skills and information. Since Jefferson's time, publicly 
financed schools and their programs have certainly proliferated, but the ^ 
American people' have clung tenaciously to that original commitmen! to skills 
acquisition and socialization. Few public schools stray far from those 
broadly conceived cduca-tional goals without hazarding a public protest. " ' 

But Jefferson's expectation' that public education would improve the 
economic, social, and intellectual lot of the citizens is an enormous burden 
for a modern educational system to bear. If disadvantaged citizens are 



iThomas Jefferson, "Letter to John Adams" (October 28, 1813) in George 
McMichael, et £l. (eds.)> American Literature , Vol. I (New york: Macraillan 
Publishing Co., Inc., 1974), p. 485. 

^The belief actually predates the formation of the State of Virginia or th 
Nation. The Plantation at Plymouth, for instance, long before Jefferson s 
time, maintained a system of "public Education," albeit largely religious in 
nature. * • 



educated at public expense and yet remain disadvantaged, the fault is usually 
ascribed to the quality of the education thej receive rather than, say, to > 
the socioeconomic system at large. But the very notion of quality in edu- 
cation Introduces the difficult problem of measurement which has racked the 
educational system for the last 20 years. How do you measure quality 
in an educational program? And, to a public that has grov/n increasingly 
chary with its funding, how do you measure cost-effectiveness? As long as 
the measure of quality was restricted largely to school resources (often no 
more than adequate facilities), the public belief that education contributed 
to the improvement of life in the nation remained undisturbed. The expense 
of public education seemed justifiable. 

In 1954, however, the Supreme Court shattered that crude measure of qua- 
lity— facilities— -if not the assumptions upon V7hich it was based. In Brown 
vl the Board of Education , the court ruled that facilities could not possibly . 
be separate and equal. Disadvantaged peoples, who first found free access to 
pubUc education following the Civil War, were still not participating in the 
economic and social well-being of the nation. The court seemed to reason 
that the fault must lie in the quality of their education. Jhe ruling was, 
in effect, an attempt by the judiciary to formulate a public policy about 
the relationship between the quality of the entire educational experience and 
the quality of life in the nation. The measure of that relationship introduced 
by the CoUrt, racial balance, was starkly quantitative even If it ignored other 
aspects of the school system. Ten years later, 98.9 percent of the Negro 
students in 11 southern states were still attending all-Negro schools and, 
therefore, as measured by racial balance, still not receiving a quality educa- 
tion^ In 1964„ Congress set out to collect data to force compliance with the 
law. The public trust in the ability of education to Improve the quality of 
personal and national life was intact if not heightened by these events. 



' The Civil Rights Act o£ 1964 commissioned a survey to determine Lh4^ , 

"lack of availability of equal ^educational opportunities for individuals by 

3 ' \ 

reason of race,- color, religion or natural origins." The" effect ^o^ Brown 
the Board of Education , then, was to force Congress, if not the -educational 
system, to conf ront="the difficult problem of meas\S>ring quality in education. 
The survey-TTsed some of the most sophisticated measurement device^ arid_ an»lyt- 
ical methods available at the time, and the ensuing report, Equality of E9%r , 
cational Opportunity (hereinafter called the Coleman Report), was an atterafxt 
to provide an accurate description of school resources and to estimate the 
effect of those resources on student ochiewement. Coleman, concluded that the 
study "has had its major impact in shif-tirtg policy attention from the tradi-: 
tional focus on comparison of inputs (the traditional measures of school qualil;.y 
used by school administrators: per-pupil expenditure, class size, teacher 
salaries, age of building and equipment, and so on) to a focus on output, and 
the effectiveness of inputs for bringing about changes in output." But the. 
results of the survey were wholly unexpected: there apgdared to be almost 
no relationship between the quality of the school and student achievement.^ 
Major reanalyses of the original data have both confirmed and denied the find- 
ings. To the public, tlie report seemed to indicate that schools did not 
affect student Achievement a great deal, and the feasy trust bcf&veeri the 
citizenry and the educational system was seriously disrupted Cor the first 
time since Jefferson proposed publicly financed schools. 



\, Gordon Mcintosh, £t ^i. , eds. , "Introduction," Equal Educational 
Opportunity (Cambridge: Harvard University Press, 1970), p. 3. 

^James S. Coleman, "The Evaluation of Equality of Educational Opportunity" 
in F. Hosteller and D. Moynihan (eds.), On Eciunlity of Educational Opportunity 
(New Yo tic: ILindQm House, 1972), pp. 149-130. - 

^Frederick Mostellcr and fianiel P. Moynihan, /a Pathbreaking Report" in 
On Equality of Educational Opportunity , 02- Sl^ > ?• 15. 
< • i " * . 

• '10 
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Cost-\ff ectiveness immediately became an ,is6ue: were the publi.C expen- 
ditu^es under>itlel^f the Elementary and -Secondary Education Acts and the . 
Headstart projects ill-advised? Was there, in fact, any relationship between 
the quality of expensive school resources and the scholastic achievement of 
the students? ' "The Coleman Report, its progeny and the stream of negative 
; Title I evaluations have gradually eroded the assumptions underlying compen- 
satory policy"^ a"n4. "cpmforted those who would reduce the resources allocated 
to education."^ • o ' * 

The Coleman Report, however, ran counter to intuitive a sumptions about 
the effect of schooling. (if a child never went to school, how well would he 
perform on a i2th grade New York State Regents /Exam in Physics? How would 
this child* s perfor^vance cotnpare to the perfomance of a student who majored 
in science and never missed a day of school frpm kindergarten to 12th grade?) 
Recent critiques of the Coleman Report have focused on the measure of student 

achievement oised in the report: was it a measure of "achievement" or rather 

.8 

of "aptitude"? (Schools can hardly be held responsible for aptitude.) The 

" *> 
difficult problem of the measurement of program resources and student achieve- 
ment has in fact become onaof the major areas of educational research since 
the, Coleman Report. ^ ^'Prior to 1964, for examplia, no more than a few hundred 
thousand dollars were spent annually on educational program evaluation. But 

9 

by 1970, the Federal Government was spending some five million dollars a year." 



^David K, Cohen and Michael Caret, "Reforming Educational Policy with 
Applied Social Res^archj" Harvard Educational Review , 45 (February, 1975), p. 23. 

7David E. Wiley and Annegret Harnischf eger, "Explosion of a Myth: Quantity 
of Schooling and Exposure to Instruction, Ma jor Educational Vehicles,** 
Educational Researcher, 4 (April, 1974), p. 7. 

"^■^^Schools do, of course, affect aptitude, but not in clearly definable ways. 
Inherited abilities arc not easily separated from abilities that are influenced 
by Gpvironment. 

^Cohen and Caret, 0£. cit . , p. 18. 
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■ 'how can the productivity of essential school programs, like reading, 
which consume -vast amounts of Federal and state £unds,^° be measured? More 
to the point, in light of recent criticism of the educational system, how can 
productivity be measured in terms o£ cost-effectiveness? The problem of 
measurment in such a productivity analysis is twofold: (1) how to measure • 
program iresour'ces in terms of dollars per hour per student, and (2) how to 
measure student achievement in terms of course content and objectives. Solu- 
tions to these two measurement problems would make it possible to determine 
the infoirmation and skills acquired during a given period of time, knowing 
the cost of each unit of time. Program efficiency would be a measure of the " 
ratio of achievement to time spent. Student achievement could then be maxi- 
mized by manipulating program resources in comparison to achievement. With 
such a continuous productivity analysis, local program managers woul^i We 
unprecedented cost- and efficiency-control over their own programs to the 
benefit o£ students, teachers, the educational system in general, and, of 
course, the public at large. ^ 

Most attempts to measure program resources heretofore have ignored a . 
major contributor to student achievement, namely the actual quantity of 
schooling. And yet no other aspect of school resources is so easily subject 
to measurement or quantification. The obvious exceptions to the general 
.tendency to ignore quantity of schooling as measured by units of time are , 
the economists, like Kiesling,^^ who traditionally use measures of time in 

lORivcrside Research Institute, for instance, has recently estimated that 
New York State alone "spends at least a billion dollars a year on reading 

related instruction." D o.i»n Concep ts for ^ ^^^'-^.^^^^ "^ TJ^ .f "^^'^^^^"g^gi^^ 
Rending! A Fcasibilin:y-FILuc]y , prepared tor blie Now VorlTlState Education 

Department (September, 197 3), p. 1. . 

• l^Hcrbert J. Kiesling, "Input and Output in California Compensatory Educa- 

tlon projects," RAND Corporation R-781-CC/RC, 1971. 

: 9 



productivity analyses* The importance of measuring time was also noted by 
Carroll dferly in the 1960* s when he deyised a model of school learning with 

the actual ^amount of time spent in learning as the major determinant of stu- 

^ 12 V ; II • 13 14 

dent achievement* More refcen'tly. Bloom and Wiley andjlaimischf eger 

have extended Carroll ' s model by incorporating school, tefaCher, and student 
characteristics (e»g», teacher sex, student age, and instructional materials)* 
When such a generalized model is refined and applied to the complexities of 
a *p articular instructional' program, however, the investigator has. to measure 
the actual minutes per week that an individual student is exposed to every 
instructional mode within the program (e.g*, \diole-group, individualized, 
and .paid-aide instruction)* 

Before such data about the quantity of instruction and other program 
resources can be related to student achievement, however, the difficult prob- 
lem of developing afi accurate measure of student achievement that is directly 
related to the objectives of the instmction must be solved* Considerable 
research has been done since the Coleman Report on the measurement of student 
achievement in reading* A clear distinction has been made between tests 
which measure student aptitude and tests \diich' treasure student achievement* 
The productivity analysis described in this paper compared Ijhe sensitivity 
of a traditional norm-referenced test to the sensitivity of an experimental 
criterion-ref erenced 1:est as measures of student achievement* The pilot 
study and the experimental test resulted in a clear conception of what a test 
must be like to measure student achievement in reading* The Bureau of School 



12john B* Carroll, "A Model of School Learning J' Teachers College Record , 
J4 (1963), pp* 723-733* 

B*S* Bloom, **rime and Learning," American Psychologist , 29 (September, 
1974), pp* 682-688* 

l^iley and Harnischf eger, o£* cit * pp, 7-12* 
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and Cultuirkl Research is cujrrently developing such a test for field-testing 
-this ^ing and fall. When validated, the achievement test will be combined 
with' accurate measures of program resources to study the productivity* of 
reading programs in New York State* ^ 
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J TradjLtional Measure^ of Student Performance 

As noted in the Introdviction to this paper, productivity analyses of 
reading programs depend upon accurate, valid measures of student achievement, 
• With rare exceptions, however, major evaluations of reading programs have* 
relied on standardized,; norm-referenced tests like the California or Metro- 
politan Achievement Tests to measure student achievement (program output) in 
reading c6mprehension» Though standardized, norm-referenced tests certainly 
have a place in educational testing>^^ there are at least five, cjj^itical pro- 
blems in using them to measure student achievement in rea^ding comprehension: 
(1) they are rigid in format and costly; (2) they are not objective enough 
in construction to be reproducible; (3) the scaling properties of the tests 
make the scores difficult to interpret; (4) the tests are insensitive to 
individual gain pr growth; and (5) they lack validity as a measure of compre- 
, hension per se « - 

Format * \ 
V Commercial firms design and develop most of the standardized,, norm- 

referenced tests that public school systems depend on. The desj.gn, construc- 
tion, and validation of these tests is time-consuming and requires consider- 

17 

able expertise (as well as what some commentators call "artistry" ), so they 
are, of course, expensive testing instruments. 

Part pf t|ie salability of these costly tests lies in their format: they 



Notably Lee H. Hansen and Karl D. Hesse, A Pilot Reading Literacy Assess - 
ment of Madison Public School Students: Final Report (Madison, Wisco'^isin: The 
MadisoD^ Wisconsin Public Schools, The Department of Research and Development, ' 
J974). . , 

^^Primarily as predictors of academic success. 

^^F.B. Davis, Educational Measurements and Their Interpretation (Belmont, 
Calif •:. Wadsworth, 1964)^ p. 262. 

•■ IT) ■' , ■ ' 
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come in pre- assembled packages that are easy to admin^iater. But it is pre- 
cisely that inflexible format which is the source of their limited utility 
and^ as a consequence, their enormous hidden costs. The rigid format, for 
instance, containing only a few jiarallef test forms, permits only one simple 
evaluation design, a pre- and a post-test. Moreover, because the pre-pack- 
aged tests cannot be^ taken apart and reassembled to construct a test of appro- 
priate difficulty for an individual student or a particular group of students, 

standardized, norm-referenced tests yield only an imprecise measure of achieve- 
ment, (in order to measure student achievement 'in reading accurately, the 
test administrator must assign the student to a test form with a level of 
difficulty which is very close to the student actual level of reading 
achievement. The more the test varies .in difficulty from the student's actual 
reading ability, the more imp4iq^$.ise the measure of that ability.) Since 

i ' ■ 

standardized, nom-ref erenced tests are inflexible in format, sin^e they 
contain few {parallel test forms, and since each form coveirs many lev^^ of 
difficulty (e.g., a 4th grade student rtay face 10th grade Wading matei^als), 
it is nearly impossible to'measure an individual stu<Jent* s reading achieve- 
ment accurately. Rigid test formats, tlien, are npt only inherently expensive, 
but they prevent school systems from implementing satisfactory evaluation 
designs* 

Objectivity , . ' 

To be useful in measuring reading achievement as part of a productivity 
analysis, a test must be objective enough to be reproducible,. That is, sev-^^^ 

eral test writers working independently v;ith the same corpus of materials 

■ * 

must be able to produce essentially the same test. What this means in prac- 
tice is that test writers, while selecting the materials to be included in 
the, tost and writing questions about those materials^ must follow a detailed, 

,16. 



explicit rules system (sSiewhat like a computer program) which radically lim- 
its the opportunity to malce subjective decisions based on personal biases and 
idiosyncrasies. Several advantages are gained by such objectivity: (a) if the 
test is reproducible, there is an objective basis for claiming that two dif- 
fcrent forms of the test should have the same label (e«g«, "readings cojnpre- 
hcnsion'Os (b) it becomes possible to examine otherwise arbitrary claims about 
what the test actually measures, for its genesis is public and traceable; (c) 
it also becomes possible to compare the results of two different tests in 
relation to the reading skills being measured; apid (d) different forms of the 
same test can be used to monitor reading development over short periods of 
time* 

• - * ■ V ■ 

Unfortunately, test development procedures for standardized tests. of 

reading comprehension fall far short of this kind of objectivity. Publishers 

liavc developed a careful, traditional procedure for constructing standard- 

izcd tests, but subjectivity is apparent at every stage of the process* Test 

writers begin, for instance, by developing an outline of the infoirmation 

the test will cover.^ ;But since "the outlining(procedure is ill-defined, it 

is difficult to verify that an item measures the content claimed by the 
18 

label." Once the^content is selected, the test writer^is constantly 

making subjective decisions ^about which questions to write. Some questions 

are rejected as too easy; others are too difficult, or too wordy, and so on. 

The result, as Bormuth has' commented, is that the test writer is "implicitly 

designing the test" as he goes along, "but doing so in a manner that is not 

19 

open 'to inspection and ..; review." Perhaps it is precisely the relevant 
— '18 

John R. Bormuth, On the Theory of Achievement Test Items (Chicago: 
The University of Chiqago Press, 1970), p. 12. 

^ Ibid., p. 13. ' , 



course content that is present in the final form of the test, but the sub-,* 

i 

stantial lack of objectivity makes verification impossible. 
Scaling Properties 

To be useful in measuring gain or cost- effectiveness, an achievement 
test must produce scores on a scale with equal intervals and a meaningful ■ 
zero point. A ruler, for instance, is a measurement device with equal 
intervals and an absolute zero. An inch at either end of a'^ilft.^^^&^till 
an inch, or an inch in linekif space is equal to ^ny other measure of one 
inch in linear space. But part of the "meaning" of that measure of one inch 
is the possibility of |i&er9/ length or no inches. The intervalv^of one inch 
is an absolute measure that does not need to be transformed ior comparison 
with ahother measure in inches. Once a test is developed to measure gain 
on a scale with equal intervals and a meaningful zero point,, it becomes 
possible to interpret the raw sc^ Is »2 true quantitgive measure of gain 
^V.^wth within individual students over, a period of tlime. 

In addition to equal interval scaling 'and a meaningful zero point, a 
useful test development procedure must be b^^pd on person-free item calibra- 
tion and item-free person measurement. Such a procedure would result in 
test scores that could be interpreted in tertns of an absolute scale (person- 
free) rother then in relation to the particular etudents who took part in 
the original calibration of the test. The procedure would also produce test 
scores. that woul^ not be dependent on the particular items used on the test 
(item- free). Person- free and item- free tests in reading comprehension would 
result in a "measure of achievement on a scale from "little ability" to "raaxi 
roym ability," Interpretation bf raw scores would be referenced directly 
to this equal interval, meaningful zero^scale. Equivalent and parallel test 

18 
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forms could then be assembled for accurate, periodic testing. School dis- 
tricts could compare the effects of different educational treatments on in- ^ .' 
dividual students or groups of students, and cost-effectiveness studies, would 

be feasible. ^ 

But norm-referenced tests do not have these scaling properties. Instead, (• 
test scores are referenced to the particular group of students used to 'norm 
the test. The estimate of reading ability these tests produce is dependent 
upon particular people and the specific content of the test. Comparisons 
between test scores on different forms of the test are made difficult, in 
part, because the conterit, of the two forms is not comparable. Raw scores can- 
not be interpreted because there is no meaningful zero point and no equal in- 
terval scale. Standardized, norm-referenced tests, therefore, cannot produce , 
accurate measures of achievement and cannot be used effectively to evaluate 
reading programs. . <j 

Sensltlvi ty ^ 

Rather than assessing gain within individuals, standardized norm-refer- 

enced tests are designed to "measure the stable, between- individual differences ' 

• 20 

•that traditionally have been of primarj^ interest to psychological testing." 
The design principles of such tests, that is, are psychometric (rather than 
eduraeSic^^); 'they deliberately maximize individual differences. For example, 
questions that most students answer either correctly or incorrectly are elim- 
inated from the t^sts in the experimental stages. The most efficient question, 
for purposes of dif f ere,ntiating between individuals, has a passing proportion 



20Ronald P. Carver, "Two Dimensions of Tests f Psychometric and EdumetrlC,^* 
American Psychologist (July, 1974), p, 512, 

^^Edumetric tests are designed "to measure the gain or growth of individ- 
uals." Ibid,, pp. 512-518, 
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of .50 (or .625 when corrected for guessing). The tests, then, arc referenced 

■ to a norm gfoup rather than to an absolute criterion; thoy are "so constructed 

22 

that at each grade level they attain a normal distribution of test results."' 
The reliability is determined by internal "^^consistency and the stability of - 
response to the, same test administered at two different times. Any sensitiv-- 
ity norm-referenced tests might have for measuring gain or growth within in- 
dividuals over the period of a schiDol year is systematically eliminated in the 
item-selection process. Standardized, norm-re ferenoed tests, that is, are 
designed to be insensitive to achievement. As a consequence, they are also 
insensitive to differences in educational treatments and cannot be use^ effec- 
tively to evaluate reading programs. I 

' ^> ■ 
Validity ' 

Standardlzedj^'torm-referenced tests are validated by correlations with 
other standardized^ norm-referenced tests. Considering the brief discussion 
of objectivity and sensitivity above, the circularity of the validation pro- 
cess is apparent! 

Moreover, there is good reason to suspect that the conceptualization of 
comprehension £er se behind these tTests is faulty. For instance, traditional, 
norm-referenced tests of reading cWprehension "purport to measure how well 
a student understands what he is reading. The questions used to ascertain 
the degree of this understanding are based on the tacit assumption that a 
direct relationship exists between reading a passage and answering questions 



22 X. 
• Harry Singer, "Measurement of Early Reading Ability: Using Nonn>Refer- 

enced, Standardized Tests for Differential Assessment of Progress in Learning 

How to Read and in Using Reading for Gaining Information,*' paper presented at 

the Conference on Early Reading Tests at Geoi-gctown University, August 20-22, 

1973, p. 4. ... 
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about it."^"^ Weaver and Bickley tested that assumption with samples from^ 
many standardized tests of reading comprehension. They discovered that col- 
lege students who had not read the passages^ upon which the questions were base 
answered 67 percent as many questions correctly as those who had the pas- 
sages,^^*. Obviously, many questions were not passage -dependent. The passages, 
that is, we^e n^ the only sources of the information needed to answer the 
questions. A moretecent study of passage-dependency by Tuinman in grades 
4, 5, and 6 found that the "average probabilities of correct responses with • ' 
no passage present ranged between '.32 and .50~well above the expected chance 
score of .25.."?^ (The norm-re§€renced tests used in this study were (a) The 
Nelson Reading Test, (b) The California Achievement Test, (c) the SRA-Achieve- 
ment Series, (d) The Metropolitan Achievement Tests, and (e) The Iowa Test 
of Basic Skills. ) These studies indicate that reading comprehension scores 
on traditional, mul tiple-clioice tests are grossly affected by the knowledge 
that students bring to the- testing situation* These norm-referenced tests 
can hardly be accurate measures of "reading comprehension" if students can 
answer significant proportions of the questions correctly without bothering 

to read the passages. 

«i . ' _ ' 

If not achievement or comprehension per se , what then do standardized, 

norm- referenced tests in reading comprehension actually measure? A test 

constructed to maximize discriminatory power will emphasize aptitude at the 



23 

J. Jaap Tuinman, "Determining the Passage Dependency," Reading Research 
Quarterly ^ Vol. 9 (1973-74), p. 208. . 
24, ' 

>/endell Weaver and A.C. Bickley, "Sources of Information for Responses 
Reading Test Items," APA Proceedings, 75th Annua l Convention (1967), 

pp. 293-294. V " 

25 

J. Jaap Tuinman, ojg. cit . , p. 206 • 
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26 

expense of achievement. In reading comprehension, such tests, emphasize 

» 

general verbal ability and 'thus correlate highly with tests of intelligence. 

In the upper grades especially,, these reading tests include more and more 

tasks like those found on intelligence tests; "hence it is understandable 

why correlations between reading and intelligence tests increase as children 

27 

progress through the grades." If "reading comprehension" tests are in 
fact measures of general verbal ability or aptitude and as si^ correlate 
highly with intelligence tests, it is not surprising that educational treat- 
ments hardly affect the results of those tests. Schools can be held account^ 
able for gains resulting from instruction, but they cannot be held responsible 
for the intelligence of the students. School systems need an accurate meafsure 
of reading achievement based upon a clear concepticprT of what reading compre - 
hension is and which aspects of comprehension can be effeqjbively taught in 
formal instruction. ^ I. 



26 

Richard Anderson, "How to Construct Achievement Tests to Assess 
Comprehension/' Review of Educational Research , Vol. 42, No. 2 (Spring, 1972) 
p. 165, 
27 

Harry Singer, 0£. cit . , p. 4. 
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>.^^^^^ New Measures of Student Achievement * 

Defining Literal Comprehension 

•]5ducati<^al systems will always be held accountable for teaching « stu- 
dents minimum reading comprehension skills, skills that will enable studentft 
to participate as adults in a literate society. (This does not imply that 
schools will or should stop teaching more sophisticated comjprehension 
skills.) Adults must be able to "comprehend" the written materials they 
encounter In their daily lives (e.g., newspapers and magazine^ employment 
information and. application forms, consumer and citizen information). 
Indeed, basic literacy has become a national prlorlty"wltness the Right-to- - 
^^ead' Program. 

At the moment, however, educational systems do not have tests which can 

ft 

diseuLangle the measure of minimum skills in reading comprehension from 

estimates of general verbal ability or intelligence*. Not on/y^ are there lio 

tests to measure minimum comprehension in %:cad.ing, but experts cannot even 

agree about the number of subskills involved in the complex phenomenon called 
28 

"comprehension." Given this state of affairs, some clarification of ter^ 



5 tninology^nd practical decisions about the number of measurable sklllsr 



/ 



comprehension^ are in order» 

For purposes of conceptualizing the process of comprehension, the 
commonly accepted definition is the logical starting point: comprehension 
is the act or process of grasping the meaning of a verbal message. "Meaning" 
ifl the key word in this def inii^i^i. A test of comprehension must ^et at the 
student's grasp of "meaning." The obvious Implication is that the test-maker 

^^F. B. Davis, "psychometric Research on Comprehension in Reading", in F» 
B. Davis (ed.). The literature of Research in Reading with Emph'a3;|s on 
Model s (U. S. Officii of . Education: Targeted Research a.nd Development Program 
OEC-0-70-.4790, ^project No* 0-9030, 1971). 



must first identify the •'meaningV' or, TROte,, generally/ the kinds of meanings . 

' ■ - . ^ .Vr , - . ■ ' # 

(e«g«, liters^l or inferential), that are to 'b^^ grasped. 

But Meaning is itsel/f a difficult concept. Broyn defines meaning, in 
its broadest sense, *'as the total , disposition to makq^u^e of and react to" 
a linguistic form* ' The re adt^r must make us^ of every aspect of a* linguis- 



tic form: *'ed," |or Sin stance^ wh^n added to the verb, strldulate , chang^^g- 
the *»meaning" of stridulateis but any further meaning that may b6 derived ^ 
from* stridulate is dependent upon the reader's knowledge of the relationship 
between stridulate and other foms of verbal and non-verbal experience. The 
cjiptionary defines stridulatej > renders it meaningful, by relating it to 
other forms of verbal experience. The meaning of stridulate finally resides, 
howe\;^, . in the relationship between the \*ord and specific kinds of behavior 
of certain insects. Stridulate , then, is. a symbolic representation of non- 
verbajf experience. In effect, context, in its largest sense— perceived 
^connections between linguistic ^otms and other foms of verbal and non-verbal 
experience— determines meaning, and context necessarily extends beyond the 
immediate group of linguistic forms in which a word may happen to appear on 

one of its occasions. The "total meaning" of a particular word would thus 

ft. 

refer to a complete history of its usage or .its symbolic representations. 
But language is not only sjnnbolicj it is also arbitrary: context, though 
necessarily expansive, is also necessarily limited or societal ly standardized 
Society, for instance, judges some contexts to be inappropriate (e.g., anil- 
ing pencil or fatly happening --»di sregarding "poetic license") and arbitrarily 



29 / 

John B. Carroll, "Defining Languag(^ Comprehension" in Roy 0. Freedle 

aild John B. Carroll (eds.), Lan^uafie Comprehension and the Acquisition of 

Knowledge (New York: John Wiley and Sons, 1972), p. "10. 
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Roger Brown, Words and Things: An Introduction to LanRua^e (New York: 
The Free Press, 1958), p. 109. 



excludes some usages from the "total meaning" of a word. Meaning is thus 
societally standardized. Even^ that limited context, however, can be enor- 
mous in the case of particular words. "Rat," for instance, may "mean" quite 
different things to "rat psychologist's" or families who live in urban ghettos. 
The first step-in defining miniinum comprehension, then, is to limit mean- 
""ing, and meaning is limited by reducing context. Carroll has suggested that 
it might be possible "to separate several 'pure' comprehens?.on factors (de- 
pending, respectively, on lexical knowledge, grammatical knowledge, and an 

ability to 'locate facts' in paragraphs) from an inferential factor requiring 

31 

the exaninee to go beyond the data given." The implication is that "com- 
prehension may be regarded as a process that contains at least two stages: 
(a) apprehension of linguistic information, and (b) relating that informa- 
tion to wider context. "^^ It is worth repeating that context, though it can 
be limited, can never be reduced to the words in any given written message; 
the words, in and of themselves, are meaningless unless they represent other 
verbal or non-verbaj. forms of experience. Even the simplest comprehension 
involves considerable knowledge of grammar and the lexicon beyond what is 
immediately evident in the message. But Carroll's point is well taken. 
Extending context by making inferences Involves an accumulated body of knov?- 

ledge as well as mental skills that are slow to develop. Schools, in fact, 

33 

may have little effect on the growth of such skills. Indeed, many students 
never develop critical comprehension skill sj they may never be able to derive 

.Blprecdle and Carroll, op . cit ., p. 8. 
3 2iyid. , p. 13. ^ 

3^Robert H. Ennis, "Children's Ability to Handle Piaget's Propositional 
Logic: A Conceptual Critique," Review of Educational Research , 45 (winter, 
1975), pp. 1-41. 



sophisticated inferential meanings from written materials, yet they can 

certainly attain "basic literacy*" . 

In light of the foregoing, minimum comprehension is conceptualized as 

literal comprehension: the act or proces-s of grasping the e^licit^ soci*^i 

etally standardized meaning of a verbal message* 

This line of argument suggests that an *adBqU£ite' 
comprehension of .a message at the time of its 
reception may be achieved by the comprehension *of 
just that linguistic information that is 'committed' 
to "the message in terms of its ovm structure and in 
tems of whatever Information has been disclosed by 
virtue of previous context*^^ 



Th 



e context, then, is restricted, insofar as possible, to the explicit state- 
ments of the particular message u'nder scrutiny* Inferential and metapho^iq^l 
meanings are radically reduced in the conception of literal comprehension 
outlined here* The comprehension factors depend on grammatical and lexical, 
knovjledge and the ability to locate "facts" in the message* Literal compre- 
hension, though limited, is not only teachable but intellectually and soci- 
ally useful, for' it is fundamental to any other derivation of meaning from 

> -S 

the message* 

For testing purposes, meaning must not only be derived, t^ut it must bo 

exhibited* A good^^test of literal comprehension would be a word-f or-xi^ord or* 

d summary paraphrase of a given message* Such tests, hpwever, are difficult 

to construct objectively, and they are expensive, even in tho multiple-choice 

format* Whot is needed is a proxy for a test by paraphrase, a test that gets 

at the student's grasp of meaning by reference to words in limited contexts 

rather than complete paraphrases* The cloze technique seems to offer possi- 

# 

bilfetes for? measuring literal comprehension objectively find ^inexpensively* 

c 

^^Frecdle and Carroll, o£* cit *, p* 12* 
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The Hgloze Technique . • 

The cloze technique is derived from studies in Gcstalt psychology on 
the human tendency to "cloze" or fill in gaps in any perceptual pattern in 
such a way as to maintain the symmetry of the perceived pattern* For exam-' ' 
plef a blank in the following series of nurn.bers— -1, 2, 3, 5, 6*— is per** 

ceived as the number, 4» Though Taylor adapted the cloze technique to the 

35 • 
reading field more thsLmSQ years ago, it has never been fully realized 

as a flexible, economical, and accurate testing device in reading. In fact, • 

the cloze technique, insofar as it has been used at all, has generally been 

used "to measure readability. In accuracy and general prescriptive value, it 

has proven to be far superior to conventional readability formulas based on 

36 

sentence length and difficulty of vocabulary. 

IrJhen the cloze technique is used* to measure readability, five to ten 
passages 250 words or more in length are randomly selected from the first, 
second, arid third sections of a printed text* Every fifth vrord is then de- 
leted, and the incomplete text is presented to the student who tries to fill 
in the missing words. Only exact replacements (minor misspellings excepted) 
lof the missing words are counted as correct answers. There is no time limit 
to the test. The student^s ability to produce the correQt words is a func? 
tion of the difficulty (readability) of the text for that particular student. 
Average scores from a group of students yield an accurate measure of the 
readability of particular printed materials for those students. Other 
readability formulas do not generally yield scores that can be directly 

^^.L. Taylor, "'Cloze Procedure*: A New Tool for Measuring Readability," 
Journalism Quarterly , 30 (1?53), pp. 415-433. 

^^T.C. Potter, A Taxonomy of Cloze Research, Part I; Reada bility and 
Reading Comprehension (Los Angeles: Southwest Regional Laboratory for Educa- 
tional Research and Development Tech. Rep. No. 11, 1968). 
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referenced to specific instructional materials and a particular student 
population. ' ^ 

Using the standard cloze technique to measure readability, however,' is 
tedious and tame-consuming* The Cesfes have to be scored by hand. Further- 
more, readability barely taps the potential uses of the cloze. For instance, 
deletion of a particular part of speech (e^g., propositions) may yield a mea- 
sure of language development. On th-e other hand, deletion of nouns, verbs, 
adjectives, and adverbs— v7ords which carry most of the informational burden 
of a sentence— i<nay produce a measure of literal comprehension as defined in 
this paper; the act or process of grasping the explicit, societally-stan-^ 
dardized meaning of a verbal message. In the standard cloze technique used 
to assess readability, any word is deleted, irrespective of part of speech, 

and the scores correlate substantially with traditional reading comprehen- 

37 " . 

sxon tests* > 

The rationale for regarding the ctoze as a measure of reading comprehen- 
sion is rooted in the regularity of language^. For exatiple, verbs are usually 
preceded by noun phrases (nouns, pronouns, and modifiers), or articles like 
"the" are usually follov/ed by nouns, adjectives, or adverbs (e.g., "the hap-, 
pily married man"). Thus a student faced with a gap in the following. sen- 
tence— "the car careened madly doi;^ the hill"— has enough syntactic 

cues to know that the .missing word has to be an adjective. The ability to 
predict words in connected discourse demonstrates at the very least a low . 
level of literal comprehension based upon grammatical knowledge. Further, 
in conve.ntional usage, some words are more likely to occur in conjunction 
with other words for semantic as well as syntactic reasons, tars careening 



37john lU Bormuth, "Cloze Test Readability: Criterion-Reference Scores," 
jfournal of Educational Meosureniont , 5 (1968), pp. 189-196. 
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madly down hxlls, for instance, are more likely to be "runaway" than, say, 
"supercilious" even tlaough both words can perfom the functions of an adjec- 
tive as required by the grammatical context. Tlius the ^loze technique also 
measures lexical meaning, another factor in literal comprehension. Know- 
ledge of factual content, the third factor in literal comprehension, is more 
difficult to measure early in a verbal message. The ability to predict fac- 
tual content (e.g., names, dates), however, increases vrith the length and 
coherence of the message. For example, it might be impossible to tell if a 
character's name were "Pickle" or "Frank" early in a narrative, or whether 
he weighed 152 or 310 pounds, but once those facts were established, they 
would generally remain consistent through the rest of the narrative* 

In general, because both the reader and the vzriter share conjmon e^eri- 

> ■ - 

ences with language, the reader is able to predict the missing words in th6 * 

v/riter's massage. The more familiar the content and the more redundant the 

style and the content, the more easily the reader comprehends the message. 

Thus the ability to predict successfully a significant proportion of th^ 

writer' s message in a cloze test is indicative of the reader's comprehension 

of the message. 

Prediction of the missing words in connected discourse, it should be 

noted^ is based on probability rather than certainty. Particular words, that 

is, are more liki^ly to occur in particular syntactic and semantic contexts 

than other vzords. Even so, guessing the missing words on a cloze test is 

similar to cafrent notions of what the reading process s^ems to be like. 

Goodman, for example, has described reading as a "psycholinguistic game"; 

Efficient reading does not result from precise per., 
. ception and identification of all elements, but from 

skill in selecting the fewest, most productive cues 
necessary to produce guesses x^hich are right the 
first time. The ability £o anticipate that which 
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has not been seen, of course, is vital in reading, 
just as the ability to anticipate vrhat has not yet 
been heard is vital in listening. 38 ' 

In a report -published in 1971, Bormuth attempted to validate the cloze 

39 V 
as a proxy for tests of reading comprehension^ Bormuth conceived of com- 

prehension as information gain, measured by comparing pre- and' post- test 

scores. Using a standard cloze f o"rmat--^assages 250 or more words in length, 

any-word deletion— Bormuth compared the cloze scores to more traditional 

/ • * 

fill-in or completion comprehension questions on the same or comparable 

' ■ . '' ■ 

passages. The relationship between the information gained' from the-'passages 

and the cloze scores is shown in FIGURE 1 as a series of curves for each 

grade level. Note that below a cloze score of 357., the^re is little or no 

information gain' at any grade level. As the cloze scores advaVice beyond ^ 

35*yt, students are able to gain infcreasing amounts bf information from tlie 

passages. The curves continue to rise steeply to a peak of about 707. on 

the cloze scores/ The average raw score on the oloze is then converted to a 

percentile figure and interpreted as follows: 

. Cloze Scores ' Reading Level 

< 0% to 347a Frustration Level Material 

J 35% to 49% ' Instructional Level Material 

50% and above Independence Level Material 

Students who score below 357. are judged non-literate in regard to the mater- 
ial in question. Those students- who score between 35 and "49 percent can gain 
information from the materials with instructional aid. Above 50 percent, 
students can gain information from the material • independently. (As noted 



^^Kenneth S. Goodman, "Reading: A Psychol inguistic Guessing Game" in H. 
Singer and R. Ruddell (eds.). Theoretical Models and Processes of Reading 
^(Newark, Del.: International Reading Association, 1970-71), p. 260. 

3 9 John R. Bormuth, Development of Standards of Readability; Toward a 
Rational Criterion of Passape Perfonivince. Final Report (U.S. Office of Educa- 
tion: Project No. 9-0237, 1971). • • . 
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FIGURE 1 
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Regression of Inf omatlon Gain Scores on Cloze Scores 
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COWPLETION TESTS 




CLOZE 



^Olbid, , p. 102. 
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above, neither conventional readability fomulas nor standardized tests of 

reading comprehension yield such clear-cut interpretations.) 

"The foregoing discussion illustrates in part how cloze scores might bo 

used to measure reading comprehension in g^iierell. What is needed in practice, 

however, is a variation of the standard clozeyfechniquc that will disentangle 

the measure of literal comprehension from mora general reading comprehension 

skills involving inference. In addition, an objective sampling technique is • 

needed so that the cloze test can be referenced directly to the materials 

that students are e^qpected^ to read at any given level in schools The tests 

must be easy and economical to construct, requiring no particular e^qpertise 

in langUtige or testir^, and allowing for the devaLopment of paral/el or equiv- 

alent test foms to monitor achievement in comprehejision over relatively short 

periods of time» Finally, fthe format must be multiple choice so that the 

tests can be scored by-machine. The cloze testing system developed by the 

Bureau of School and Cultural Research meets all these criteria. 

The Cloze Testing System Developed by the Bureau 
— ; — ^ 

Selection of Cloze PassaV,es« In order to eliminate the possibility of 
content bias and to assure that these tests would reflect the materials stu- 
dents actually read in instructional ptogrants and elsQwhere, a pl^n was de«- 
vised for random sampling of four domains: • 



Im Textual Material in Reading, L/inguage Arts, 
Social Studies, Science, and Mathematicsj 

2. Citizen Material (newspapers and news magaztines); 

3. Constimer Material (catalogs, 'advertising, instruc- 
tions, and so forth); and 

\ ^ 4. Reference Materials (test instructions, children's 

magazines, encyclopedias, and so forth). 

The sources of these materials x^ere the New York State Education Depart- 
ment's Curriculum Laboratory and the State Library. Some consumer passages 



were ^aken from A Pilot Reading Literacy Assessf^lfefc of Madison Public School 
Student s » 

The samples taken were examined to locate coherent passages of specified 
lengths appropriate for dozing. Readability scores were then calculated and 
the passages assigned to grade levelsi (As noted in the preceding section 
of this paper, the cloze is itself superior as a measure of readability to 
the conventional formulas described below. Consequently, conventional read- 
ability formulas were. used only as rough, preliminary guides for placing the 

passages at appropriate levels of difficulty^ for field-testing.) 

42 43 
The readability fonnulas used were the Spache and tlje Dale-Chall» 

i . . . 

The Spache is normally used for grades 1 through 3, the Dale-Ch^ll fof^grades 

\ through 12 and college. Both formulas use average sentence leiig^^ and per- 
cent of "hard words*' in calculating difficulty. "Hard words*' are those not 
appearing on lists of familiar words. The v/ord list for the Spache fonriula' . 
is ••clarence Stone^s Revision of the Dale List of 796 Easy Words"; for the 
Dal,e-Chall formula it is the "Dale List of 3000 Familiar Words," (The 
criteria for difficulty used in devising both' fonnulas were graded reading 
materials.) The Spache formula produces grade level scores. The Dale-Chall 
formula produces raw scores Interpreted as "corfected grade levels.*' The 
corrected grade level for a raw score of 5.0 to 5.9 on the Dale-Chall, for 
example, is 5th to 6th ^rade. 



^^Hansen and Hesse, cit . 

^^G. Spache, "A New Readability Formula for Primary-Grade Reading 
Materials." Elementary School Journal , 53 (1953), pp. 410-413, and Good 
Reading for Poor Readers (Champaign, 111.: Garrard press, 1960). 

^^E. Dale and J. Chall, "A Formula for Predicting Readability: Instruc 
tions." Educational Research Bulletin, 27 (1948), pp. 37-47. 
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The range of Spache scores was divided into six equal intervals, and the 
range of Dale-Chall scores w^s divided into 22^equal Intervals. This gave 
28 difficulty levels covering grades 1 through college. The raw scores, 
difficulty levels, and original grade level interpretations given by Dale- 
Chall and Spache are shown in TABLE 1. 

The selection procedures resulted In" 1,494 passages vf or cloze testing. 
Their distribution by domain and grade love! is shpwn In TABLE 2^- TABLB 2 
also shows the distribution of the textual materials by subject matter. 

Preparation of Cloze Items . The procedure for word deletion in th.e 
cloze passages varied with grade. For grades 1 and 2, every eighth word was 
deleted. Deletions were limited to nouns and verbs. For grade 3 and above, 
every fifth word was deleted. Deletions included adverbs and adjectives as 
well as nouns and verbs. , 

In all cases, the initial deletion was made between the sixtOi and the 10th 
words. The exact starting point -^as determined by a table of random numbers. 
The number of deletions per passage was fixed by t*lie passage length, which 
varied by grade levAl. The number'of alter?natives in the multiple-choice 
responses also varied by grade level: three alternatives at gr^de 1, four 
at grades 2 and 3^ and five ^t grades 4 and above. These variations by grade 
level are summarized in TABLE 3, Specifications for Cloze Passages and Test 
Items. 

The correct multiple-choice response to a cloze item is the exact. word 
deleted from the passage. To assume, distractors of appropriate difficulty 
for the test items, graded lists of nouns, verbs, adverbs, and adjectives . 
were prepared using Harris and Jacobson^s Basid^ Elementary Readipg 
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TABLE 1 

Difficulty Levels for Cloze Passages 

0 * 



Headability 
Founula 

c . 


Raw Score 


Dit txcuity 
Level' 


Original Grade Level 
AssigTinent: by Spache 
and Dalo-Chall 


s 
p 


!• 0 - 1. 4 

1 «i - 1 0 


1 

2 


1 . 


A 
C 


2. 0 - 2. 4 
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TABLE 3 * * • ^ 

Specifications for Cloze Passages and Test Items 



Grade 1 



Grade 2 



Grade 3 



Grade A 
and Above 



Passage 
Length 



25-35 
words 



40-45 
words 



60-70 
words 



60-70 
words 



Ir/ords 
Deleted 



Nourrs 
verbs 



Nduns 
verbs 



Nouns 
Ad jec* 
tlvcs 
Verbs 
Adverbs 



Nouns 
Adjec- 
tives 
Verbs 
Adverbs 



Frequency 
of Dele- 
tions 



Every 

8th 

Word 



Every 

8l;h 

Word 



Every 
5 th 
Word 



Every 

5th 

Word 



Deletions 
per Pas- 
sage 



10 



10 



Altcma'tiive^ 
per Item 



4 
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Vocabularies ^^ and p:DL Research and InformaLion Bulletin 5; A Rgvlsed Core 
/\ Vocabulary , Special content lists of v;ords for subject matter areas like 

Social Studies were compiled using the Harris-Jacobson material and the Amor'* 

,46 • 
' lean Heritage Word Frequency Book * 
— ^ ^ • 

Initially, distractors were selected from appropriate lists by use of 
a table of\ random numbers. Later, a computer program was written for auto- 
matic raiWom selection of distractors. Each set of distractors was revieue^ 
to eliminate tricky or ineffective distractors, such as synonyms, and to 
assure^that the distractors agreed with the stem in tense, number, and so 
forth. 

With a minimum of 3 deletions per passage at grade 1 and a maximum of 
10 deletions per passage at grade 4 and above, nearly 15,000 multiple-choice ^ 
items have been prepared for the cloze testing system. 

Format . All cloze passages and test items have been put in a comparable^ 
fotmaU,^<^e format gives (1) the identification number of the passage, (2) 
a title (provided by the item writer), (3) the passage itself, and (4) the 
test Items. Large (Bulletin) type is used for the first two gradeS;^ A 

r 

sample cloze passage for grade 8 is shown in FIGURE 2. 

Advantages . The multipldtchoice cloze testing system being developed 
by the Bureau has several important advantages over standardized, norm-refer- 
enced tests of reading comprehejision. Its primary advantage Hes in its 
specif icity --rather than attempting to assess a confusing hodgepodge of 



^^A. Harris and M. Jacobson, Basic F.lcmentary Reading Vocabularies (New 
York: The Macmillan Company, 1972). 

^*5s. Taylor, H. Frackenpohl, and C. White, "A Revised Core Vocabulary; 
A Basic Vocabulary for Grades 1-8, An Advanced Vocabulary for Grades 9-13," 
Research and Informj^r^ion BulIcLin No> 5 (Huntington, N.Y. : Education Dcvelop- 
ment Labbral;ories, 1969). 

Carroll, P\ Davie s, and B. Richinan, Tho American Heritage Word Fre- 
quency Book (New Yark: Houghton-4^Iif f lin, 1971). 
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FIGURE 2 

Sample Cloze passage and Items 



THE CHEST SAVES THEIR LIVES 



The wind blew freshly, and drove the \ 

away from the shore, and the 2 . . • billows tossed it 

^3^ and down; while Danae 4 her child 

5 



closely to her 

6 wave would dash its 



, and dreaded that some 

7 crest over 



them both. The 
9 



8 



sailed on, however, and neither 

nor was upset; until, when 10 was 



coming, it floated so near an islemd that it got entangled in a 
fisherman's nets. 



© 



© 



0 



0 



a. instrtiment 

b. register 

c. energy 

d. chest 

e. variation 

a. uneasy 

b. special 

c. indigo 

d. light 

e . husky 

a . conveniently 

b. really 

c. exclusively 

d. up 

e. highly 

a. stitched 

b. clasped 

c. halted 

d. mooed 

• e • tangled 

a. atlas 

b. bosom 

c. Jcnock 

d. application 

e. applicant 



o 



0 



© 
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a. 


next 


b. 


big 


c. 


fluffy 


d. 


legal 


e. 


racial 


a. 


moral 


b. 


remarkable 


o.. 




d. 


stranded 


e. 


heroic 


a. 


concord 


b. 


patter 


c. 


chest 


d. 


wrist 


e. 


push 


a. 


whisked 


b. 


sank 


c. 


returned 


d. 


educated 




entitled 


a. 


thing 


b. 


maize 


c. 


menu 


d. 


night 


e. 


' clearance 



cognitive skills, the multiple-choice cloze Is designed to measure a care-f 
fully delimited aspect of reading, literal comprehension. Not only is lit- 
eral comprehension vital to more sophisticated Interpretations of verb^il 
* messages, but it is construed In this paper as very close to notions like 
"basic literacy," thereby. establishing its social as well as its academic val 
ue* By m^ans of the multiple-choice cloze testing system, the definition of 
literacy can be referenced directly to whatever materials school districts 

decide their students ought to be able to read. "Basic literacy" can thus 

•t ^ 

be defined in tems of specific reading materials. The possibilities for 

'flexibility on the one hand or uniform standards on the other are evident. 

Two characteristics of the design and development o£ the testing system 

make it possible to detemine whether or riot the tests actually measure lit- 

I ^. 

eral comprehension: objectivity in cotistruction, and the passage-dependency 

' of the items. Passage-dependency is inherent in the design of the testing 

system. Without reading the passage, the student has little basis for choos- 

Ing aaong the graded lists of, nouns, verbs, adjectives, or adverbs that coiii- 

prise each set of responses. For example, how would a student Identify the 

correct answer in the following responses: 

. ' . . * a. >Aiisked 

» ' b. sank 

^ ^ c. returned 

" ' d. educated 

* ' " e. entitled 

And yet If the student can read the passage and understand the semantic con- 
text surrounding the missing word, he or she should have little difficulty in 
choosing sank as the correct answer: **rhe chest sailed on, however, and nei- 
ther nor was upset" (from FIGURE 2). 



*Amore complete description of the actual construction of the testing 
system Is available on request from the State Education Departoient, Divis- 
idn of Research, Bureau of School and cultural Research, Albany, Now York 
12234. 
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In contrast with norm-rcf erence^^ests that purport to measure "reading 
comprehension," the multiple-choice cloze testing system is designed to mea»» 
sure achievement rather than general verb^ aptitude. The tests, that is, 
are sensitive to gains \7ithin individuals rather than differences between 
indxviduals. The test scores ax*e an accurate assessment of the acquisition 
of specifiable kinds of meanings from tipecific kinds of printed texts. The 
tests, therefore, should be sensitive to differences in instructional treats* , 
ments of both short and' long duration. 

This precision in measurement results in the second major advantage of 
the testing system-- its flexibility . The testing system can be used to esti 
mat^ an individual students' gain in literal comprehension of specific ±n*^ 
structional materials over relatively short period^s of time, or it can be 
used to compare the efficiency and cost-effectiveness of different reading 
prograns. V7ith modifications, the testing system can be used to measure read 
ability (for placement purposes) 5 language development (for diagnostic pur- 
poses), and either literal or general comprehension. 

But this kind of flexibility depends on a scaling system with equal 
intervals and a meaningful zero point. In addition, the testing system must 
yield estimates of achievement that do not depend o^^the particular passages 
or items used, or the specific students used to scale the difficulty of the 
passages. In order to accomplish the^ objectives, a modification of the 
Rasch procedure will be used. ^' 

To calibrate the difficulty of the passages, two. sets of passages and 
items containing some passages and items in common will be given to two 
different groups of ^students. The tw sets of passages will then be calx- 
Brated separately. "The resulting calibrations will represent two scales 
with equal measurement units but different origins because the origin of 

41 



the [passage] calibration scale is arbitrarily set equal to the aver- 

f- -I 47 

age Lp^ss^8®J easiness*" The two different scales will then be' ad- ^ 

justed by adding a constant to the difficulty value for each passage on the 

less difficult test* This will result in one scale origin for both tests* 

48 * J 

An example helps to clarify the procedure* Assume that Test A and' 
Test B share three passages* The average difficulty value of the three pas- 
• sages on Test A is 55* On Test'B, the same thtee passages have an average 
difficulty value of 45* To adjust the calibrations in order to equalise the ^ i 
scale origins, it is necessary to add a value of 10 to the difficulty value 
tof the passages on Test B* This 'procedure references all passages to the 
same scale of reading achievement* The result is person-free passage cali- 
bration* (This same procedure can be used to scale reading passages across 
-subject areas like science and language arts*) 

Another advantage of these procedures lies in the ability to anchor t^he 
scale of reading achievement at low levels of literal comprehension* T>is 
can be accomplished because the passages in the multiple-choice testi^ sy$- 
tem have already been roughly scaled for difficulty using the Spache and ^ 
Dale-Chall readability formulas* This scale provi4ds an initial reference 
point for low difficulty passages* Through several applications of the Rasch 
scaling procedures, it will be possible to anchor the comhion origin for these 
passages at a meaningful zero point (l*^e*, the easiest passages that students-^ 

■ ■ ^ ^ . 

with the lowest comprehension ability can comprehend)* 

^^R* Robert Rentz, "implications of the Rasch Model:. Test Equating and 
Score Interpretation," Educational Rei^arch Laboratory, University of Georgia 
(photocopy)^ p*4* ^ 

^^his example is taken from R* Stiles, W* Hathaway, and F* Foster, "Use of 
the Rasch Model with Content Referenced Tests," paper delivered at the Ameri- 
can Educational Research Association Annual Meeting (Washington, D* C*: 
March 31 -April 3, 1975), p* 2* ' ' ' 
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Thus the objectivity of test construction^ and the scaling procedure make 
it possible to produce numerous parallel t^st forms for continuous monitoring 
of achievement in literal comprehension or productivity anal^^es^ The test- 
ing system is extremely flexible, and flexibility is the key to the tliird 
major advanta^i^of the multiple-choice cloze testing systenif»-its economy ♦ 
One system, satisfies a variety of the evaluative needs of school districts. 

Validation ^ The Bureau is currently field testing the multiple-choice 
cloze ie^ts., ^nd the field tests will continue through the 1975-76 academic 
year* A concerted effort will be made to detemine whether or not the exper- 
imental tests can in fact separate the measure of literal cqmprehension from 
more sophisticated inferential processes* The testing system is designed to 
measure the student's grasp of the explicit, societally standardized meanirig 
of the printed, verbal message* Performance on the multiple-choice cloze 
tests will be compared to performance on several other Sasts designed to liiea-f 
sure various aspects of reading comprehension* 

The preliminary investigation of^ the validity of the multiple-choice 
cloze "tests was initiated in the spring of 1975*. In this new study, perfor- 
mance on the experimental cloze tests will be compared to performance in 

1* sub-tests or s^^ected items from the California 
Achievement Test (e*go, items that measure 
grammatical, lexical, or inferential meaning), 

. 2* a cloze test where* students have to supply the 
missing words, and 

3* a criterion-referenced test (described below) 

designed by the Bureau especially for validating 
the multiple-choice cloze test* 

The assumption in comparing the multiple-choice cloze tests to specific 

J.tcm^s oh the California Achievement Test is that the experimental cloze will 

have a high po sit ivei correlation with those items on the norm-referenced test 

that measure grammatical and lexical meaning* On the other hand, relatively 

, . 43 
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lower positive correlations arc objected with those itcjms on the California 
Achievement Test that involve inferential processes and higher levels of read 
ing comprehension. 

■ J 

Since the California Achievement Test is. a standardized, norm-referenced 
test, it has a limited utility for validating the multiple-choice cloze sys- 
tem» Consequently, the bureau developed another criterion-referenced test 
comprised of passages v^ith main idea and detail questions tp help clarify 
what the multiple-choice cloz'e test actually measures. The Bureau set up / 
specifications for the passages in tenQs of length, difficulty of style and 
concepts, variety in^style and subject matter, quality (e.g., coherenpe), 
and Reader interest, and thfen selected and edited passages from existing 
tests^ books, and magazines. In addition^ many, passages were written to the 
test specifications. The process resulted in 300 passages at 20 levels of 
difficulty from grades 1 to 10, or 15 passages per level. The passages were 
assigned to levels of difficulty using the ssme Spache and Dale-Chall read- 
ability formulas that v;ere used to place the multiple-choice cloze passages. 

Main idea and detail questions were written on each of the 300 pacsa£;es, 
using an e3q)licit rules system to reduce writer bias. The title and main 
idea questions necessarily involved inferential processes, but inference was 
kept to a bare minimum by limiting title and main idea responses to e>qplicit 
words or statemeilts in the text. 

The detail questions— who, what, which,^ when, where,' why, and how~v7ere 

' 49 
adapted from Bormuth's On the Theory of Achievernjent Te^t Items. The ques- 
tions were based on minimal transformations of clauses from the passages. , 
The exact wording of the original statement in the passages was reproduced in 
sofar as possible in the question. For example, "obsequious men write tricky 



^^Bormu^h, op. cit . 
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questions" would be transf omed into "v/hat kin^ of men v/rite tricky ques.- 
tions»" Distractors had to be grammatically and semantically plausible and * 
were also taken directly fr oni ytihc passage whenever possible* FIGURE 3 is 
an example of a passage and questions* ' ^ , ' , 

Of the three factors involved in literal comprehension— grammatical and 
lexical knowledge and the ability to "locate facts" in a verbal message— this 
test will certainly measure the first and third factors* The example above 
should make it obvious, however, that it is possible to answer the question 
without/ knoxdlng what "obsequious" means* Therefore, the test may not be 
an adequate measure of lexical meaning* Vocabulary and V7ords-in-conteXt 
items from' the California Achievement Test will be used to validate the mea- 
sure of lexical meaning on the detail questions. 

The overriding criterion for every item x^as that it be impossible to 
answer the question without reading tjie passage* Consequently, passages with 
a high *Bgree of familiarity (e*g*, the Columbus story) were eliminated* In 
addition, items without passages will appear on Part II of the test forms to 
verify pas^^e-dependency* * * 

During t|ie preliminary study of validity conducted in the spring of 
1975, only the detail questions (the least inferential) \rLll be compared to 
the multiple -choice cloze test* Additional studies of validity will talce 
place tlurin^ the fall and spring of 1975-76* At that time, the multiple- 
choice cloze test will be compared to other tests of literal comprehension 
(e*g*, paraphrase) and inferential processes* 

The Test Development Notebook * Following the field-test and resultant 
refinement of passages and items, the multiple-choice cloze testing system 



*A more complete description of the construction of the test is Avail- 
able on request from the Bureau of School and Cultural Research. (See 
address at bottom of page 33,) 
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FIGURE 3 

Sample Literal Comprehension, Main Idea and Detail 

A l-onc swordsman was quietly eating his dinner* in a small Japanese 
inn. He appeared calm, not bothered by the four flies buzzing around 
him. Three ronin (samurai unattached to any of the great feudal clans) 
enLered. They noticed the magnificent swords in the man's sash, and the 
old, faded clothes that shpwed he too was clanless. They knew the swords 
were worth a small fortune, and they confident that one man was no 

match for the three of them. 

The three men sat dox-^n at the next table and began to make loud 
corrnnents ahout their neighbor, hoping to g6ad him into a duel. He seemed 
to take no notice and their remarks became ruder and more pointed. The 
swordsman merely raised his chopsticks; in four quick snips, he effort- 
lessly caught the four flies on the wing. As he slov/ly laid down the 
chopsticks, the three ronin hurriedly left the rooi:^. They had( tried to 
pick a fight v/ith Miyamoto Musashi, one of the. greatest sv7ordsmen in the 
history of Japan. 

1. The best title for this selection is 

A, A Lone Swordsman 

B» A Small Japanese Inn 

C. A Small Fortune 

D. Feudal Clans 

2. The main idea of this selection is 

A. The 'three ronin hurriedly left the room. 

B. The three ronin noticed the magnificent swords in the man*s sash. 

C. The three mejLsat down at the next table and began to make loud 
> comments about their neighbor. 

D. The three ronin had tried to pick a fight with Miyamoto Musashi, 
- *onc of the greatest swordsmen in the history of Japan. 
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3. How did the swordsman appeal;? 



A. calm 

B. ' confident 

C. bothered 

D. rude 



4# Vlliat became r^der? 



Ai the three men*s manners 

B. ' the three men's remarks 

C. the three men's language 
D# the three men's behavior 



VJho knew th<* swords were worth a snail fortune? 

A, a lone swordsman 

B, Toshiro Mifune 

C, the three ronia 

D, Miyamoto Musashi 



6. What had the three renin tried to do? 

A. kill the swordsman 

B. leave the room 

C. enter the inn 

D. pick a fight 

7. Wlien did the three renin hurriedly leave the room? 

A. before they had finished their dinner 

B. after they had finished their dinner 

C. as the swordsman slowly laid down his chopsticks 

D. when the. swordsman sl.owly drew his long sword 
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will be used in productivity analyses of reading programs* In addition, 

■ ' ' \ 

the passages and items will be made available to teachers in the fom of a 
Test Development Notebook* The purpose of the Notebook is to provide suffi»» 
cient numbers of test passages and items at various levels of difficulty s6 
\^J^ that literal comprehension at a given level may be monitored periodically 
with the administration of series of comparable test fonxis* In its final 
fom> the Notebook will be a computerized bank of passages and items from 
which any number of tcst;s can be constructed at specific levels of diffi- 
culty in particular subject areas* 
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Empirical Investigations to Dat e 
fleasurinR Program Resources 

As noted in the Introduction to. this paper, considerable research ctfort 
has been conmitted to the measurement :of student achievement in readiiij; com- 
prrehension since the Coleman Report. More specific and useful conceptions of 
comprehension have resulted from that concerted effot-t. In addition, theo- ' 
rists like Carroll have helped make important distinctions between eencr'al 
tests of verbal ability on the one hand and tests of comprehension per se on 
the other. Out of these research efforts ^^and conceptual distinctions have 
come the first tentative tests of student achievement in literal comprehension. 

» 

But the development of accurate measures of reading comprehension, though 
e difficult and important research goal for the bureau, is not an end in it- 
Self. What school districts need is a practical arid economical system for ana- 
lyzing productivity in reading programs* The ultimate goal, then, is not only' 
to measure achievement in reading comprehension accurately but to help school 
districts increase student achievement without raising costs . Before the new 
achievement tests can be used to study * productivity In reading programs, how- 
ever, researchers must solve the vexing problem of measuring program resources 
accurately. 

Not only is there a bewildering array of program variables to measure 
(e.g., teacher age, sex, degree status; whole-group. Individualized, paid-aide 
instruction; student body characteristics; facilities), but the complexity of . 

0 

the measurement problem is further confounded by the need to quantify the ac- 
tual use rather than the mere availability of program resources. In his study 
.-of Title I programs, Cohen indicated the need to study the use rather than 
the availability of resources. He found, for instance, that there is at least 
as much if not more variation between classrooms in the way resources are used 
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as there is between different programs. Measures of program resources, 
therefore, roust be accurate enough to reflect the variations in the use of 
resouirces between classrooms. Moreover, in order to analyze the impact df 
resource utilization on student achievement, precise measures of resource 
allocation must be directly related to individual students within the class- 
room. 

Productivity studies and related research have generally avoided these 
practical problems by using levels of analysis well above the Individual stu- 
dent. However, researchers have identified a viable unit of measurement for 

i • 

t \ ■ ' 

program resources. 

Gamer has recently suggested that "to measure the actual utilization 
of the various resources means nL more than to measure the tteie in which they 
are employed in various ways. In a fundamental sense, allocation of resources 
(human or material) is allocation of time."^^ If the problem is accurate mea- 
surement, time is a variable of central importance in studies of school re- 



sources, for >^ 

Time can be measured with as vnhi. precision as the , \ 
researcher desires. The measurjes of time have pro- ^ 
perties that ar« almost impossi/ble to secure in our 
conventional measures of academic achievement: equality 
of units, an absolute zero, and clear and unambiguous 
comparisons of individuals. Furthermore, time as a 
variable can be put into economic and resource costs 
for the individual learner, for groups of learners, 
* and for .schools and communities. 

Over the last decade, researchers have developed several improved mathe- 

\ 

SOoavid K. Cohen, "Politics and Research: Evaluation of Social Action 
Programs in Education," Review of Edu cational Research, 40 (April, 1970), pp, 
213-238. / o e 

SlwiUiam Garner. "Inputs and Outputs in the Educational Process" (ED-075- 
9A8), a paper given before the American Educational Research Association Annual 
Meeting in New York (February 2-7, 1971), p. '6. 

52b, s. Bloom, "Time and T.o«,.r,4»g." American PsvcholoRist, 29 (September, 
1974), pp. 683-684. ^ 
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matical models of school learning which allow for the conceptualization of a 
very complex process in terms that arc eminently quantifiable- CarrolUs 
model, for instance, relates the time A student needs to learn a task to 
the time actually spent learning the task: 

^ * ^ /-/ time actually spent ^ 
Degree of Learning « f( time needed 

In an elaboration of Carroll's model, Wiley and^Harnischfeger noted that 
"achievement ... is directly determined by only two variables: total time jieed- 
ed by a pupil to learn a task .. • and total time a pupil actually spends on a 
given task."^^ The model, as expanded by Wiley and Hamischfeger, becomes 

Achievement «= f (^^ ■ ) 

where 

W is the total Allocated Exposure Time, 
X i§ the percent Active •Learning Time, ... 
Y is the percent Usable Exposure Time, [and] 
Z is the total Needed Learning Time.^^ 

The parsimony of this equation is attractive. The practicality and util- 
ity of the symbolic conversion of student and program characteristics Into 
time factors, however, must be verified empirically. The real complexity of 
school learning behind the Wiley- Hamischfeger equation is indicated by the 

conceptual model in FIGURE 4. When such a generalized model is applied to 

J ■ ■ • 

S 

53john B. Carroll, "A Model of School Learning," Teachers College Record, 64 
(1963), pp. 723-733; "Problems of Measurement Rfilated to the Concept oC 
Learning for Mastery." Educational Horizons , 48 (1970), pp. 71-80; and Fitting 
-a' Model of School Learning to Aptitude and Achievement Data Over Grade Levels 
(RB-73-51) (Princeton, N.J.: Educational Testing Seirvice, 1973). 

^\iley and HarniischEeger, op. clt .,, p. 11. . 
^ ^Ibid ., p. 11. 
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Total Usable Exposure Time 




* 3. 


Tolal Active Learning Time ^ 



n . Achievement 



56 



Ibid . , p. 10. 
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the complexities of a particular instructional program, however, the inves- 
tigator has to measure the actual minutes per week that an individual studei}t 
is exposed to evpry instructional mode within the program (e.g., whole-group, 
individualized, and paid-aide instruction). The pilot study reported below 
incorporated both conventional measures of resources* and time as measures of 
resource utilization* 

A Pilot Study of Productivity in Reading Programs 

• The Bureau conducted preliminary research on the feasibility of a pro- 
ductivity analysis of reading programs in 197A-75* An exploratory study was 

0 

designed to develop and validate a practical and economical method for analyz- 
ing reading programs at the local leVel» The major research questions, in 
light of the previous discussion of achievement tests and the measurement of 
program resources, were framed as follows: 

i» Is a criterion-re'ferenced test of reading comprehension 
more sensitive to variations in the allocation of program 
resources than a standardized, norm- referenced test? 

2, Do educational production functions (the formal statement 
of educational input-output relationships) differ f6r 
school districts? 

The preliminary analysis was based on a sample of 3047 students in 
grades 4, 5, and 6 in four school districts* These school districts, identi- 
fied as A, B, C, and D in TABLE 4, were roughly comparable in student socio- 
JABLB 4 

Characteristics of School Districts* 





Variables 




District 




A 


B 


C 


D 




District Type 


Suburban 


Urban 


Urban 


Urban 




% White 


89 


83 


42 


63 




7. Lower Income • 


25 


36 


59 


37 




No. Pupil^s 


567 


947 


1010 


523 




No, Schools 


2 


7 


5 


3 ' 




No, Teachers 


36 


56 


60 


25 
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economic status. District A, however, had higher percentages of upper in- 
come families and white students in each class, while District D had lowei: 
percentages of upper income families and white students in each class. 

These particular school districts were selected because they varied in 
the way they used program resources to teach reading. District A, for exam- 
pie, had a modest investment in added resources for reading instruction, pri- 
marily in reading centers run by specialists who performed a coordinating 
function. The other three districts had more extensive investments in com- 
pensatory reading programs which were primarily independent adjuncts to the 
regular programs. 

Since the focus of the study was the individual student, it was neces- 
sary to quantify factors that might affect a student's achievement in read- - 
ing. As a consequence, the' Bureau gathered data on more than 70 school and 
program variables in each district. (The Technical Appendix* contains a com- 
plete list of all the variables on which data were gathered.) Following a 
complete datj, edit, datn with low reliability wfere eliminated. A listipf the 
38 variables included in the main analysis reported in this paper is given 
in feiBLE 5. 

A unique element of this research project was the estimation of ins true - 
tional time actually allocated to individual students within classrooms and 
in adjunct compensatory treatments. The Bureau demonstrated that it is pos- 
sible to gather reliable, data on instructional time for large numbers of in- 
dividual students efficiently and economically. Data on the quantity of 



*Tho Technical Appendix to this report is available on request from ^he 
St..te Education Department:. Division oC J^cscorch, Bureau of School and Cultural 
Research, Albany, New York 12234. 
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TABLE 5 ^ 

Variables Ii|»cluded in the Main Analysis 



Variable 
Number 



Variable Name 



1 Student's age ^ 

2 Father's occupation 

3 PEP raw score in 3rd grade 

4 Ho. of pupils in reading 

class ■ • -^^j^fl: - 

5 Teacher's degi^p^tatus 

6 IMRU (Index o£ in^tferials 

resource utilization). 

7 Post CAT^ Total Reading 

8 Effect for school, 1^ 

9 Effect for school 2 

10 pffect for school 3 

11 Effect for school 4 

12 Effect for School 5 

13 Effect for school 6 

14 Teacher's age 

15 % white in class 

16 % working poor 

17 % unskilled 

18 k % skilled blue collar 

19 % skilled v/hite collar 

20 % business 

21 % professional 

22 Mi.nutes per y6ar (mpy) 

xAole^group teacher 
instruction^ 



Variable 
Number 



Variable Name 



23 Mpy small-group teacher 

instruction 

24 Mpy individual hetp by 

teacher 

25 individualized instruc- 
tion, teacher 

26 Mpy small-group instruction 

27 Mpy individual help 

28 Mpy individualized instruc- 

tion V 

29 l^y total, teacher instruc- 

tion \ 

30 Mpy specialist instruction 
33. Mpy piaid-aide instruction 

32 Mpy unpaid-aide instruction 

33 Pte-CAT Total Reading 

34 P re-CAT by whol e-group 

instruction^ 
'35 . Pre-74 CAT by stolL-group 
instruction 

36 P re-CAT by individual help 

37 Pre«CAT by individualized 

^ instruction 

38 Classroom Socioeconomic 

Status Index 



^Eventually deleted and replaced with variable number 38, Classroom 
Socioeconomic Status Index* 

^Pupil Evaluation Program: standardized, norm-referenced reading and 
math tests developed by the N6w'York State Education Department. These tests 
arc administered in grades three and six« Only grade three tests are used in 
this report. • • 

c 

California Achievement Test (1970 norms): another standardized, norm- 
rcforcnccd test. As many as three levels and two forms o^ this test were 
used. 

^'his is simply an overall control variable representing the gross effect 
"Of each school within a district. 
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^All time variables were natural log transformations of minutes per year. 
^All interactions involved variables in standard z- score form. 
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reading instruction were gathered using taped interviews with principals, 
teachers, Specialists, and selected teacher aides. (The Technical Appendix ^ 
contains the interview schedules and instructions* )^ 'fTlie teacher and special- 
ist interviews resulted in estimates pf the actual minutes per week per stu- 
^dent of reading instruction in each of four instructional modes: whole-group, 
small-group, individualized, and individual help. These multiple interviews 
verified the actual time allocated to each student for reading instruction in 
each of the four modes. Time estimates in these modes were also obtained for 
additional instructional time by paid and unpaid aides, the minutes per 
we6k per student were converted into minutes per year^ resulting in a more 
accurate -measure of resource allocation by accounting for student absences. 

0 

In addition to instructional time estimates, the interviews provided a 
record of all materials and equipment used as resources in teaching reading. 
An index ©£ materials resource utilization (IMRU) was developed to quantify 
simultaneously the^ type of instructional resources used by a teacher and the 
extent of this utilization. The interview schedule grouped instructional 
resources into the following categories: (1) basal series, workbooks, and 
other skill-builder supplements; (2) additional software; (3) hardware (equip- 
ment); and (4) teacher-created materials. To obtain an BIRU for each teacher, 
a total score for all categories was based upon the number of materials used 
in each category and how they, were used. Materials used as a major resource 
were given a value twice that given supplemental materials like "additional 
workbooks.'' The final IMRU was thus the sum of the scores for each category 
; of matTerialSr 

In this experimental study, the analysis differs substantially from the 
approach suggcs^iicd by Wil-ey and Harnis,ch£cgcr in that there was no attempt 
made to convert all school and program variables into time factors. Instca'l, 

■ : ■ '■ ' no .... 
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all input variables were analyzed independently and simultaneously. This 
independent and simultaneous analysis resulted* in a much simpler conceptual 
model as indicated in FIGURE 5. . ^ 

In accord with the major categories of the conceptual model, Additional 
research questions were framed as follov^s: . 

1. ' Does the amount of time allocated to teading instruction 

^ affect reading achievement? . . 

' ^. . ■ ^ . ' 

2. Does the amount of time allocated to different instructional 
modes (whole-group, small-group, individualized, individual 
help)- affect reading achievement? 

"3. Does the availability of instructional materials affect read- 
ing achievement? , 

4. D<3es student ability as measured by norm- referenced tests 
affect reading achievement? 

5. Does teacher quality (e.g., experience, preparation time, 
training) affect reading achievement? 

6* Does the number of students in the reading class affect 
reading achievement? 

7. Do individual school characteristics affect reading 
achievement? • 

8. Does the affect of instructional time on reading achieve- 
ment vaty as a function of student ability. 

As indicated in the conceptual model in FIGURE 5, the allocation of pro- 

gr^am resources was compared to test scores on both criterion- and norm-l^^afer- 

enced tests. The California Achievement Test (CAT) and the Pupil Evaluation 

Program (PEP), ^^developed by the New York State Education Department, repre- 

sented standardized, norm- referenced tests in^the pilot study.' Raw scores 

for previous reading ability were used from tHic PEP. Achievement Development 

Scale Scores (ADSS) from the CAT were especially useful because they had 

interval scaling and normal distributions. In addition, the ADSS were inde- 

pendent of form, level, grade, time of administration, and restandardization. 

The bureau developed the criterion*-referenced test especially for the 
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FIGURE 5. 

Simplified Conceptual Model of a» Reading Program 



INPUTS 



INST,RUCTIONAL PROCESS 
OR READING PROGRAM 



oirrpuTS 



Im Instructional Time 



r 



(eog»y minutes per 
week [mpw] of read- 
ing instruction) 



2* Instructional Mode , . , ■ — 
(e*g*9 mpw of small 
group in s t rue t ion) 

3. Instructional Materials 
(e#g»9 number of 
basal readers) 



4# Individual S tudent^Xharac- 
teristics — , — ■ ■ 



(erngmy entering perfom- 
ance) 

5. Individual Teacher Charac- 
teristics . ' ... ■ ' ' — 



(e»g»> years of experi- 
ence) 

6m Student Body Characteristics 
of Reading Class 



( e* g* 9 average soc ioeconomic 
status of class) 



7» School Characteristics 



iefgmy the overall school 
effect)^' 



8. Interactions pf Inputs 



(e*g* 9 instructional time 
with entering achievement) 



. 1. Student Reading 
/ Achievement on 



the Criterion- 
Referenced Test 



— ^ 2. Student Reading 
Ach icvement on 
the Norm-Refer- 
enced Tests 



*These are control variables representing the effect of a particular 
school within a district. * ^ 
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pilot study. Committees of teachers and reading specialists in each of the 
nine participating school districts selected from the Bank of Reading Objec- 
tives* those> objectives in the, areas of vocabulary and comprehension that 
seemed most pertinent to their own rdading programs. Lists of basic sight 
words and of graded reading materials supplied by thd school districts were 
used as a guide in the selection of passages ?nd vocabulary items. 

Grades 4 through 6- had been selected for study. Criterion-refer- 
enced tests, howevei;, must be geared, to the level at which students are actu- 
ally achieving, irrespective of their grade placement. Since the range of 
achievement was approximately seven years for these three grades, seven level 
of the test, correponding roughly to grades 1 through 7, were constructed 

The longitudinal defeigij of the study called for repeated criterion- refer 
enced measurement on the same students at regular intervals. All school dis- 
tricts administered the criterion-referenced tests five times at 2- week in- 
tervals be&^een March and June of 1974 as indicated in TABLE 6. Five equiva- 

TABLE 6 

Design for Data Collection 

'■ p^il May February March April May June 

Data Instruments ^^^^ ^973 1974 1974 1974 1974 1974 



PEP X g 

CAT X 

CRT 

Reading Program Data 

Gathered via Inter- ' 

views and School 

Records \ \ 



XX XX 



BThese data on the CAT were available for a sub-sample of students. 

*Compiled by the Bureau, the Bank contains some 2000 objectives organized 
into six areas: Multisensory Readiness, Decoding, Vocabulary, Comprehension, 
Location and Study Skills, and Reading in the Content Areas. In the pilot 
study, items were restricted to two areas— vocabulary and comprehension. 

' \ no 
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lent forms of the test were constructed ^t each level, and students were 
assigned in random order to the various forms in such a fashion that ultimately 
every student took all five forms of the test at the level to which he had 
been assigned by bis teacher* i 
Analytical Procedures 

The analytical procedures were designed to provide evidence about the 
researxih questions. The procedures included data editing, reducing the num- 
ber of variables, and deriving the parameters of the reading programs* Follow- 
ing the data edit, frequency distributions were obtained for all variables. % 
if 

(VThen there were gross ami>unts--157. to 20%--of data missing for a variable, 



it was eliminated.) The relatively skev:ed distritgtions led to the conver- 
sion of all time variables into natural logarithms. Means, standard devia- 
tions, and correlations were than calculated. Data with low variability were 

eliminated. ^ 

A principal components analysis with yarimax rotation was then completed 
on a large proportion of the raw data matrix, including multiplicative inter- 
actions of instructional time and quality variables. The results from this 
factor analysis led to a reduc^fllh in the number of original variables and 
to the conclusion that student aptitude was interacting with instructional 
time. Therefore, in subsequent analyses, each students' third grade PEP scores 
and midyear CAT scores were used as statistical controls on ability. The de- 
pendent variable was either the CRT or the norm- referenced JunG^CAT, Vajriables 
for the interaction of instructional time with ^tudent aptitude were included 
in some of the analyses, but these analyses are only reported in the Technical 
Appendix. 

With the number of variables reduced to a manageable set, a series of 
production functions was developed. The research questions focused on 

CO 
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estimating the relationship between the input and output variables listed in 
FIGURE 5. A production function is a formal statement of these input-out- 
put relationships and can be expressed symbolically as 
* . Y « f(xj, x^) 

where 

Y == student performance on a CRT or NRT, 
== instructional time, 

== instructionat"^ materials, 

== teacher *degree status. ' ^ 

. In this case, final student performance is a linear function of time, 
materials, and teacher qualifications. In the case of interacting inputs, 
the least complex relationship could be expressed as Y = f(x x^) where final 
student performance is a function of instructional time multiplied by teacher 
degree status. The form of this relationship may be such that a student's 
performance will improve with more instructional time by a more highly 
trained teacher* 

The Importance of the program inputs in determining student achievement 



was arrived at analytically through mult^le regression analysis. A more 
formal account Illustrates the procedures used. Beginning with the input 
ydriables as predictors-- that is, X^, X^— and a criterion variable, 
Y, the procedure assumes a linear model of the genei:al form, 
Y = ^X^ + b2X2 + b3X3 + b^X„ + c 

where Y is the performance measure predicted on the basis of the "n" predic- 
tors^ weighted by the numbers b^ b^ plus an additive constant "c." The 

b-weights ate raw regression coefficients. The "sign" (i.e., + or -) and 
magnitude of the b-weights are Important. If the effects of the X's 

■ ci ■ 
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(i.e.>, the inputs) on Y (i.e., student performance) are linear, then a 
positive sign means a positive relation exists between an input and student 
achievement. A negative sign usually indicates a negative relationship; for 
example^ if instructional time increased, student performance would decrease. 
The magnitude of the b-weights can be interpreted as follows: if an input (X) 
changes one unit, the output (Y) changes "b" units. For example, if b = .50 
for X, then as X changes one unit, Y changes half a unit. Usually, the effect 
of a particular input variable is not interpreted unless its associated b- 
weight is statistically significant. The statistical significance of the 
b-weights is determined by reference to the "t" distribution. The direction 
and magnitude of the input or resource variable in predicting student achieve- 
ment are summarized in the following section of this paper. 



* 

Results 



As indicated in FIGURE 5, a conceptual model was developed as a frame- 
work for dealing with the niain questions of interest. These questions were? 

1. Is a ,criterion- referenced test of reading com- 
prehension more sensitive to variation in the 
allocation bf program resources than a stand- 
ardized, norm-referenced -test? 

2. Do educational production functions (the fonral 
statement of educational input-output relation- 
ships) differ for school districts? 

As an introduction to differences across districts, TA3LE 7 provides 
the means and standard deviations of the major variables used in subsequent 
analyses. For ease of reference, variables are listed under the major cate- 
gories of the conceptual-an&lytical model. TABLE 7 reflects wide 



*For a detailed mathemntical presentation of the findin^rs, refer to the 
Technical Appendix. 
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in the study sample. The teachers' average ages, for instance, varijss from 
district to district* It is encouraging to noto that the instructional time 
variables also reflect considerable diversity. Each district is allocating 
their i^i^dlng program resources in unique patterns. These patterns provided 
the basis for further analyses 9f the relationship between program resources 
and student performance. Multiple regression was used to determine the nature 
of these relationships. 

Sensitivity of Student Performance Measures . ^^^70 main analyses were 
completed in order to compare the sensitivity of the norm-referenced test 
(NRT) to the sensitivity of the criterion- referenced test (CRT). 

The first analysis determined the effects of instructional time and 
student characteristics on the CRT and the norm- referenced California Achieve- 
ment Test (CAT). The main findings indicaft%d that instructional time wa^ 
related negatively to both the CRT and the CAT. The occupation o£ the stu- 
dent's father was positively related to the CAT. ' 

Several characteristics of the first analysis suggested the need for a 
-^lightly different approach. These analyses were based on different CRT and 
NRT student subsamples. In addition^ instructional time was measured in min- 
utes per week, and student performance on the CAT was measured in grade equiva- 
lent scores. The negative time effects from the first analysis also needed* v 
further clarification. 

Thus, a second analysis of CRT-NRT differences was initiated. This anal- 
ysis differed in several v/ays from the first. Instructional time was esti- 
mated more accurately as minutes per year based on minutes per day times the 
number of days each student actually attended school. The CRT-NRT compari- 
sons were based on identical students, not separate sub-samples. Another 
major difference was in the number of statistical control variables. These 
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second analyses were very conservative regression models controlling for the 
following variables: number of pupils, percent white, and socioeconomic com- 
position of the reading class; student's third grade PEP scores in reading, 
student's age and father's occupation; teacher's age and degree status.* The 
CAT scores were Achievement^ Development Scale Scores with equal intervals and 
normal distributions. These multiple regression analyses were designed to 
clarify the extent to which CRT* s and NRT's measure different aspects of stu- 
dent reading performance. The inclusion of the control variables also indi- 
^jated relative test sensitivity to program resources. ^ 

The findings from the second CRT-NRT analysis can be summarized as 
follows: 

1. A greater number of variables defining the instruc- 
tional environment were related to CRT performance 
than to NRT perf omancc. 

2. Ins^tuctional time voriables were significantly 
« related to CRT performance more often than NRT 

performance. * 

3. Both the criterion- referenced test and the norm-, 
referenced California Achievement Test measure 
unique aspects of reading comprehensioiv. 

The greater sensitivity of the CRT versus the NRT for the instructional 
time variables must be interpreted cautiously. Many of these tljne effects 
have proven to be curvilinear, not linear. (See the Technical Apfpendix for 
details.) Detailed studies of these time effects reveal that schools differ 
in their distribution patterns for instructional time. In essence, lower 
perfoming students on the average receive more instructional time. However, 
student performance does not increase until a considerable amount of instruc- 
tional time is given to the students. Student performance also varies with 
staff type (e.g., paid aide). 

The findings on test sensitivity provide evidence in support of the use 
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of CRT's over NRT' s in studies of educational production functions. 

District Production Functions * An educational production function de« 
scribes mathematically how school inputs ^resources) relate to student por- 
formance (outputs). In these alialyses, Inputs included schpol, teacher, 
and student characteristics while output was measured by the California 
Achievement Test (CAT). The norm- referenced CAT v;as used instead of the CRT 
due to better reliability, distribution characteristics, and the fact that 
interval scaled scores could be used within each district* v-. 

These production function studies included all of thd control variables 
used in the CRT-NRT comparisons plus a control for school ef fects#^ In addi- ; 
tiOT£ to determining the importance of these variables, the Index of Materials 
Resource Utilization w£is studied. These analyses helped determine the rel- 
ative importance of the explanatory and control voriablesr in predicting stu- 
dent reading performance. 

The general findings from the production function studies for each of 

the four school. districts can be summarized as follows: 

• 1# A large proportion of th^ variance in CAT per- 
formance was accounted for in each district; 

2. The variables that significantly accounted for 
CAT variance differed somewhat from district 
to district. (For example, percent white in' 
the reading class v/as significant and positive 
in two of the four districts.) 

3. Tl^e Index of t^aterials Resource Utilization was 
not significantly related to CAT performance. 



4. Instructional time variables v;ere significant 

in three of the four districts. (In one district, 
£pur out of eight time variables were signifi*- 
cantly related to student performance.) 

5. The importance of different teaching staff changed 
from district to district. (For example, the 
teacher effect was significant and positive in one 
district. However, the effects due to reading 
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specialists atid paid aides were negative ^.n 
three districts and usually curvilinear. These 
curvilinear effects™ for specialist's time mean 
that lower performing students receive more 
instructional time and that student perfor- 
mance does not go Op until rather high levels 
of instructional time are reached. However, it 
cannot be concluded that these specialist effects 
would occur without regular classroom ipstruc- 
tioin. The latter condition did not occur in 
the study.) " * * 

4^ ■ 

The production function studies support the use of time as. a measure 
resource utilization in the schools. This fact combined with the greater 
sensitivity of CRT' s 'over NRT's produced a foundation for more legitimate 
productivity studies. Future productivity studies will measure program 
resources in units of time and^ program outputjs with criterion-referenced 
tests scaled as precisely as time. 



G7 



Conclusions and Projected RcajKarch 

The lialcyon day.s of. educational funding sccni to be gone forever. Public 
schools arc harried by fiscal cutbacks and face a future of continuing bud- 
getary strictpresi Ironically, while funds shrinkf there is a growing 
dc3Tvaj)4-^*^publ it: and prof essional— for increased quality in school programs, 
especially in critical skilly areas like reading. Talk of education account- 
ability, of value received for dollars spent; and of input-output relation- 
ships has groiTO steadily in volume in the last 10 years. In competition 
for limited funds, school programs are being forced to prove their value. 
And they are e^ec^ed to show improvement in student achievement irrespec- 
tive of funding. The critical problem facing school districts, then, is 
how to increase student achievement ^without raising costs. 

The v/ork of the Bureau of Scliool and Cultural Research reported in this 
paper addresses this critical problciii in two vjays: (1) by investigating 
evaluation in reading and attempting to design a more appropriate and 
efficient testing system, and (2) by studying the feasibility of productivity 
analyses of reading programs. 

As mentioned in the Introduction to this paper, considerable research 

effort has been committed to the measurcnvitnt of student achievemenj|; in 

.i> 

reading since the Coleman Report, but that effort has generated controversy 
as well as new concepts and tests. Proponents of nonn- and cifiti'^ftrion-rcf er- 
Gndfcd tests fire critical volleys at each other in the professional journals. 
Though the nevj criterion-referenced tests have not replaced traditional, 
norm-referenced tests, two points have cleatly emerged from the critical 
debate: (1) because noim-i'eferenced tests are, in effect, designed to be 
insensitive to Instruction and because their scaling properties make it 
difficult to interpret or compare scores, norm-rof eronced tests are not 
useful for evaluating reading ptogrms; and (2) the global measures of 



••readiixg comprehension** that characterize norm- referenced tests no longer 

satisfy the evaluative needs of school districts which are moving toward 

Individualized instruction and mastery learning. , 

In light of these considerations, the bureau set out to compare the' 

sensitivity of a criterion- referenced test of reading comprehension with 

that of a norpi-ref erenced test of reading and to study the feasibility of 

productivity analyses of jading programs by comparing the results of both 

tests to variations in the allocation of program resources. The findings of 

a pilot study conducted in the spring of 1974 indicated clearly that the 

criterion-referenced test was more sensitive to the use of program resources 

than the norm- referenced test* " ' / ^ ' ^ 

The bureau was encouraged by the sensitivity of the rudimentary, 

criterion-referenced test u6ed in the pilot study, but the inadequacies of 

..the test were just as obviotis* like traditional, norm-ref erenced tests, the 

experimental, criterion- referenced test attempted to assess a wide spectrum 

of cognitive skills* Though the items were clearly identified in relation ^ 

to particular course objectives, each test fom contained only a few items 

on' any one objective* Furthemore, neither the passages nor the items were 

accurately scaled for levels of difficulty. Finally, the test was not sxif- 

ficiently objective in construction to allow for the \economical production 

of large volumes of items for a great number of parallel test forms. 

The bureau decided to develop an accurate, useful, and economical test 

♦ 

' of comprehension per se * An attempt was made to disent^angle literal compre- 
h&nsion— the act or process of grasping the explicit, societally standardized 
meaning of a verbal message--f rom more sophisticated, inf erentiaj^..pj^cesseS 
that school systems hardly seem to affect. Literal? comprehension, as defined 
above, is very close to popular notions like *'basic literacy" and can be 
referenced directly to whatever verbal messages school districts decide their 
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students ought to be able to comprehend*. The bureau's conceptualization 
of literal comprehension, then, is not only academically. useful, for 'it is 
fundamental to any further derivation of meaning from a verbal message, but 
it is also socially useful. The public, for instanc^ expects high school 
graduates to be "literate," to be able to derive e^licit, societally stan- 
dardized meanings from reading materials like riev7S]|apers, consumer directions 
or job application forms, ^ , 

To test literal comprehension accurately and ecc^homicafly across a- wide 
array of reading materials, the bureau developed a muitipld-choice cloze 
testing system rather than a test, as such. The testing %ystem is composed 
of a pool of graded, "clozed" passages with sets of altsernate responses for ' 
each deletion. Tests can be assembled from this pool in a variety of ways 
to suit particular evaluative needs. The procedures used to grade and "cloze 
the passages arid select alternate responses were made systematic enough to 
be reproducible. (The objectivity of item construdtion is indicated by the 
fapt that the selection of alternate responses has already been computer- 
Ized.) To date, nearly 150y0 passages have been sampled from a variety of 
reading materials., ili specified domains,* graded for difficulty, and "clozed."- 
Nearly 15,000' sets of alternate, responses have been written and reviewed. 

Validation and re^nemetet of the testing system began in the spring of 
1975 and will continue through the 1975-76 academic year. In the course of 
the validation, the passages will" be calibrated accurately for difficulty on 
a scale with equal intervals and a meaningful' zero point using a modified * 
Rasch procedure, (Such a scale is a prerequisite to the production of com- 
parable test 'forms or legitimate productivity analyses.) , Eventually, the 
graded and. "clozed" passages and sets of alternatives will be stored on com- 
puter tapes. Teachers will then be able to request tailor-made test forms- 
sampled from specific kinds of reading inaterials at spej^yiG level s of dif- 



ficulty--and the test will be assembled entirely by m^?^|^g>, 
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The accurate, economical assessment of student achievement, however, 
is only part of a productivity analysis qf reading programs. It is equally 
important to measure the allocation of pijrogram resources on a comparable 
scale. In the pilot study of the feasibility of productivity analyses of 
reading programs conducted in the spring of 1974, the bureau demonstrated 
that it is possible to gather large quantities of data on the allocation 
of instructional time in the classroom and in compensatory programs. 
Measuring resource allocation in units of time is important because costs 
can then be exTpressed in terms of unit costs (i.e., dollars per hour per 
student). A cost- tiifle ^ratio is a precise, corpmon scale on which all pro- 
grams can be compared to determine relative differences in expenditures. 
Moreover, the measurement of program resources in units of time results in 

a scale with eqi^p^fe^ntervals and a meaningful zero point which can be com- 

1^' ■ - - . 

pared to the measure of student achievemant outlined above. Legitimate 

productivity analyses thus become possible. • 

Following the validation of the multiple-choice cloze testing system 

and the calibration of the passages, the testing system will be ready for 

use in a general demonstration of the analytical approach to productivity 

in reading programs. A statewide survey bf productivity in reading programs 

is proposed. A proportionate "sample of classrooms in grades 1, 3, 6, 5, 

and 12 will be drawn from a representative set of 200 school districts in 

the State of New York. The surve^y sample will number about 10,000 students 

and will allow for generalization of results by region and for the whole 

state. 

This survey will assess the amount of instructional time actually 
allocated to individual students in the regular classrooms and in any adjunct 
or compensatory programs. In addition to the allocated time, the effective 

71 

-64- 



o 

ERIC 



learning time for each student will be estimated^ Cost data associated 
with staff, facilities, equipment, and instructional materials will also be 
gathered. Further, the design of the survey will make it possible to assess 
organizational features of programs and emphasis on particular objectives* 

The overall intent of this component of the survey is to assess the 
contribution of the quantity and quality of instruction to achievement in 
literal comprehension. In another component of the survey, the impact of 
reading time and material s. in the home on achievement in literal comprehen- 
sion will be assessed* The focus on reading in the home would be relevant 
to the f ormulat:iOj[i of policy about parent- or community-oriented programs 
designed to improve reading performance* It may well be that careful in- 
vestments in the family or community context will be more productive in 
improvirtg reading performance than some compensatory treatments applied 
in the school* ' . 

Overall, the survey .nay have a number of policy^ relevant findings* 
Representative statewide estimates of levels of literacy in particular 
domains of reading materials will be available for the first time* For 
example, the survey will attempt to answer questions like the following: Can 
12th grade students comprehend the written messages on the front pages of 
newspapers? Of equal Importance will be data describing how the amount of 
time spent on reading varies in school programs -and in homes* Aside from 
this useful descriptive data, the results of the "survey will yield important 
findings on the effect of various instructional variables on reading achieve- 
ment* -It is expected that these results will make important contributions 
in the formulation of policy concerning the most productive uses of funds 
for reading instruction in New York State* 
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1 »0 Introduction 

This Technical Appendix to "Measurement and Productivity in School Read- 
ing Prograins^ — Main Report" provides a detailed stmimary of procedures, results, 
and survey instruments.^ The Appendix is v/ritten for the research special- 
ist interested in the Main Report !s corlclusions. It is designed to clarify 
procedures and results and to present studies which will help reduce dis- 
agreement over some of the research approaches employed. 

The Technical Appendix not only supplies supporting evidence for the 
Main Report, but also contains a supplemental study,' entitled "Effect of 
Comprehensive Achievement Monitoring on Student Learning." This study is 
related to the Main Report because it deals with the effectiveness of a 
similar criterion-referenced testing system. In the Main Report, the criter- 
ion-referenced testing component of the research design was embedded in a 
longitudinal design using the California Achievement Test (CAT). The Main 
Report compares norm- and criterion-referenced tests; this related study 
analyzes the same criterion-referenced testing system over a three year per- 
iod with comparisons with the Metropolitan Achievement Test, another nonn- 
referenced test. It is encouraging that this report concluded that criter- 
ion-referenced tests are more sensitive indicators of student achievement 
than norm-referenced tests. The author noted that "If educational decision 
makers want evaluation data on the effectiveness of their locally developed 
curricula, they are more likely to detect program strengths and weakness 
employing criterion-referenced measures than standardized norm-referenced 
tests." (p*25) This intact study supports the conclusions of the Main Report. 

^his Technical Appendix was written by Steven J. Kidder and Ronald E. 
Streeter. 
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2>0 Methodoloffl' 

Sample > The original student sample consisted of 5887 students in 'fth, 
5th, and 6th grades in 19 school districts. The individual student was the 
imit of analysis* Study samples were selected from the original sample for 
detailed investigations. For example, tfie number of students in the criter- 
ion-referenced analyses varied from 153 to hK^. In contrast, the number of 
students in the norm-referenced analyses went as high as 2231. 

The district-by-distriqt sample characteristics are provided in Table 
2.1. A fifth district was left out of these studies due to extreme scores 
and few students. The districts were chosen for wide variation in resources 
used in reading instruction and variation in socio-economic composition. 
All of the 4th, 5th, and 6th grade teacheirs in each school were sampled. / 
These teachers had volunteered to participate in the study and in the instal- 
lation of an experimental criterion-referenced testing systeip. 

Table 2.1 

Sample Characteristics for Each District in the Analyses (N=300'f) 



Variables District 





A 


B 


c 


D 


No. Pupils 


567 




967 


, 523 


No. Schools 


2 


7 


5 


3 


No. Teachers 


36 


56 


60 


25 ' 


Type District 


.Suburban 


Urban 


Urban 


Urban 


% White in class 






12SI5 


63^ 


% Lower Status in 










class * 


25% 


36^ 


59^ 





Research design. The basic J'esearch design for the study is presented 
in Table 2^2. The longitudinal aspect' of the design is evident in the 
repeated data collection points beginning before 197'** Final data collection 
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points occurred in June, 1^7^. The earliest collection occurred when each 
student v/as in third grade. The first data point was based on, the Pupil 
Evaluation Program (PEP) norm-referenced test in reading developed by the 
Bureau of Pupil Testing and Advisory Services at the State Education Depart- 
ment, Albany, New York. 

In May, 1973, a sub-saTiple of about I6OO students received the Califor- 
nia Achievement Test (CAT). In January, 197^i and June, 197^, approximately 
2500 students received the CAT. Of these students, I60Q had also received " 

CAT in May, 1973. Therefore, with regard to norm-referenced measures 
on all students, the final analyses varied somewhat but usually involved the 
use 01 the June/ 197^, CAT as the dependent variable with the 3rd grade PEP 
test and the January, 197^, CAT test as statistical controls for prior stud- 
ent performance;* 

The develophient and implementation of the criterion-referenced testing 
component of the design were as follows. In the development phase a j^^Jjk of 
Reading Objectives (BRO), consisting of some 2000 objectives grouped into 
six areas (multi-sensory readiness, decoding, vocabulary, comprehension, 
location and study skills, reading in the content areas), was compiled by a 
team of reading research and curriculum experts. To date, efforts have been 
concentrated in two of these areas, vocabulary and comprehension. From the 
objectives for vocabulary and comprehensioijjf. ^a^h of the nine cooperating 

• -A !' 

school districts selected those which were modfe relevant to its reading pro- 
gram, determined the grade level or levels at which each objective should be 
taught and tested^ and indicated the relative importance of each objective 
by designating the number of test items to be constructed per objective. 
The complete test was designed to be administered in a period of 30 minutes. 
In order to allow for continuous monitoring (a major goal of the project), 

81 



five equivalent forms of each test were constructed, and administered at 
intervals of two and three weeks between March and June, 197^- Subjects 
were randomly assigned to the five forms of the test. Initially, the study 
focussed on grades k through 6, but since the range of achievement within 
these three grades was considerably greater than three years, tests were 
constructed at seven levels, corresponding roughly to, grades 1 through 7 ^ 
each student being assigned by his teacher to the leval at which hte was 
achieving. 

A data-processing system was used to complement the testing program. 
The feedback provided to the Schools included group data on each item, each 
objective^ and total scores at each level. In addition, every student 
received information on his own performance on the same measures. The total 
system, which is known as Comprehension Achievement Monitoring (CAM), has 
several advantages: (1) it permits diagnostic evaluation of individual and 
group strengths and weaknesses on specific skills, thus enabling the teacher 
to deploy instruction time more effectively; (2) it allowa for the continuous 
monitoring of every student on each skill, and is therefore a useful tool in 
the individualization of instruction; (3) it facilitates flexible ad hoc 
grouping for the teaching of specific skills on which identified students 
need further tutoring; (k) as additional data are gathered, correlational 
studies should indicate empirically which of the skills are indeed most 
important to the criterion of reading comprehension • CAM is evaluated in 
the study by Adams in Section 10.0. 

The repeated critorion-referenoed testing of student reading performance 
during the spring of 197^ included a form of longitudinal matrix sampling. 
This testing procedure involved the random assignment of a set of five par- 
allel test forms to any given student using 5x5 Latin Squares such that 



an even and representative distribution students per form per class was 
obtained. These five parallel tests v/ere used in each of, 6 to 8 levels for 
grades k through 6. Thus, up to eight sets, of test forms were p3j*ovi^ed to 
^a district incoi*porating the range of reading, levels in grades four^ 'five, 
and six. Typically, a district used 3 levels and 15 test forms to monitor 
readinjg comprehension. 

During the testing schedule, large quantities of student test data 
were generated. For example, within the entire study, 3500 to ^000 students 
took five criteribn-referenced tests and two norm-referenced tests. Thus, 
over 25|000 test forms had to be processed. The data monitoring and process- 
ing system used for handling the CRT data was the CAM system. The Califor- 
nia Achievement Tests*were ^chine processed and merged (by student) with - 
the CRT and program resources data. 

The norm-referenced measure of reading performance was the the Cajiifor- 
^ nia Achievement Te^^t — ^9>70 norms- Raw, grade equivalent, and achievement 
dev^opment scale scores (ADSS) for vocabulary and comprehension were avail- 
able for each student on both pre-^d post -instructional administrations <%f 
the test. The ADSS were especially useful because of the following features: 

interval s^saling, normal distribution-, and' independence of form, level, 

. , ^- ,. ■ . 

grade, time of year, and restandardization. Thus, the ADSS can be added, 

subtracted, and averaged regardless of the level or forrn of the CAT adminis-^ 

tered. ^ 

The data needed to specifically characterize each reading program were 

gathered via interviews^^wij^h teachers, specialists^aidesi and principals 

* ■ , ■ A 
* and searches of school records. This Secured form March, ^197^i t^o June, 

197^- During the initial interviewing, the schedules were improved. This 

result-ed in the acquisition of quite detailed instructional time data on-^each 
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particular student. As* rioted, all of the detailed progrA.,jJkta were refer- 
enced directly to the individual student. Thi's guaranteed using the student 
as the imit of analysis within a school. Therefore, most of the analyses 
are based on varigitions pf individual student scores about average scores ♦ 
for students in each district. 

Interview schedules were developed for principals, specialists, and 
teachers {See Section 11.0). These schedules wefe designed for gathering 
as mucj;i information as possible on resource allocation and program config- 
uration in reading. Each schedule required approximately sixty minutes for 
administration^ Tti^ principalis schedule includied school-level questions on 
student-body SES, reading level, ethnix:ity plus detailed questions on facil- 

4 

ities and special reading programs. The specialist's schedule requested ^ 
detailed information on program materials, mode of instruction, and the time 
each specialist or aide spent with each student. The teacher schedule 
focused heavily on the classroom reading program, teacher characteristics, 
and student body characteristics. The hardware and spftware used for read- 
ing instruction, was listed. Of special interest was the tabulation for each 

student of /the actual number of minutes spent in* reading instruction witn 

ft 

teachers or specialists, or tutors in the following modes: (1) whole class 
instruction, (2) small group instruction, (3) individual help (on a one-to- 
one t^asis), and (k) individualized instructicgi (i.e., a ''mini-curriculum" 
per student, individually or in, groups).* ifi-f ' T 

• • ^ ' l' 

In an effort to combine all of ttie in^^tFuctional material variables 
present in any given reading prograin, tfie^Index of Materials Resource Utili- 
zation (IMRU) was ^developed|^^!l?he IMRU is based upon the type and arjjj^unt df 
materials used in a reading "ffeogram in & specific teachers' classroom and 
whether they were used as a majoS or suppl^emental resource. 

84 ■ . 
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The instructional materrials variables were grouped into four categories, * 
on(^for each type of material used: 1) basal series, v/crkbooks, and other 
*skill builder supplements, 2) additional software, 3) hardware, and k) teacher- 
created materiatls. .A scor'e for each category was determined; this scoro was 
based upoii the number of materials used in that category ^d how they Were 
used. In most cases, materials used as a major resource v/ere given a value . 
twice that given materials used to supplement reading instruction. The IMRO 
was determined by taking the sum of the four scores derived for each category 
of materials..^ A^„brief description of - each of the four scores making up the 
IMRU follows: 

Materials Category . This score for basal series, workbooks, 
and other skill builder supplements was perhaps the most com- 
plex. For each basal series used, a value of 2 was added. A 
value of 1 was added for each workbook used in conjiinction with 
a basal series. In addition, a value of 1 was added if one to 
three additional skill builder supplements v/ere used, and a value 
of 2 if more than three of these skill builder supplements were 
used. The highest possible score allowed for Materials Category 
# 1 was 12. 

Materials Category #2. < Additional software was grouped according 
to the number of obviously different resources used: less than 
3i 3-6, and greater than 6. Values of 1, 2, and 3 were assigned, 
respectively, when each group of different resources was used as 
supplemental resources. These values were doubled for gorups 
used as major resources. If more than six major resources of 
softwajre and ©ore than six supplemental resources were used, a 
total maximum of 9 was assigned. 

Materials Category #3* In general there were nine different types 
of hardware used. A value of 2 was assigned to each type of hard- 
ware used as a major resource, while 1 was assigned to each type 
of hardware used as a supplemental resource. The highest possible 
score, the case in which all nine types of hardware used as major 
resources^ ivas 18. 

Materials Category #k» The score for teacher-created materials is 
similar to that of hardware. Values of 2 and 1 were assigned to 
each type of teacher-created material used, depending on whether 
it was a major or supplemental resource, respectively. Since 
there were five types, the highest possible score was 10. 

As mentioned, the IMRU was the sum of these four scores. The highest 
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possible score that could have been assigned to a particular teacher was ^9. 
This would represent a comprehensive use of resources and materials by a 
teacher. 

^This research strategy provided a data base o| sufficient detail and 
focus for dealing with the major reseeurch questions of interest. 
3*>0 Original Variable Definitioxi 

The school factors listed in the conceptual -analytical model in the 
Main Report are major, superordinate categories of related variables. These 
major categories (e.g., teacher- characteristic^) are composed of specific 
variables. These variables and thrf.r possible interactions provided the 
diata base for the analyses of program effectiveness. Therefore, in order 
to provide the reader with a complete imderstanding of which variables were 

used in the study, they are listed here under their respective higher-level 
categories of school and school-related factors. 
INPUTS; 

J- 

A. Instructional Time Variables for each student included the number 
of minutes per week of 

1. Total Reading Instruction. * , 

2. All 'Whole Group Reading Instruction. 
3» All Small Group Reading Instruction. 
^. All Individual Help in Reading. 

5» All Individualized Reading Instruction. 

6. All Specialist Reading Insturction. 

?• All Paid Aide Reading Instruction-. 

8. All Unpaid Aide Reading Instruction. 

9. Whole Group Instimction by the Teacher ^ 
10. Small Group Instruction by the Teacher. • 

8G 
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11* Individual Help by the Teacher. 

12* Individualized Instruction by the Teacher. 

B. Instructional Mode Variables indicated whether a student received 

15. Whole Groi;^ Instruction by the Teacher. . 
Small Group Instruction by the Teacher. . 

15. Individual Help by the Teacher. 

16. Individualized Instruction by the Teacher. 

17. Total Whole Group Instruction. 

18. Total Small Group Instruction^ 

19. Total Individual Help* 

" 20. Total Individualized Instruction. 

C. Instructional Materials Variables^ v/ere subsumed in an index 
(Variable Number 21) based upon the type and number of materials 
used for reading instruction in the teacher's classroom. 

< 

D. Individual Student Characteristic Variables included 

22. Age. 

23. Sex. 

2^. Birth Order. 

25* Father ^s Occupation. 

26. Father's Education. 

27. Mother's Occupation. 

28. * Mother ' s. Education. 

29* 3rd Grade Reading Ability (PEP TEST) as a raw score. 
^0. 3rd Grade Math Ability (PEP TEST) as a raw score. 
31* Number of Days Absent. 

32. Percentage of Days Present. 

33. Membership in a Specific Reading Class. 



^See Section 2.0 for a thorough description of the Index of Materials 
Resource Utilization. 
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3^. Membership in a Specific School. 

55* Student raw score on criterion-referenced test^ 
1st administration at Levels 2, 5» 5» 6, 

56. Student Achievement Development Scale Score on the 
May, 1975 and January, 197^» California Achievement 
Test for vocabulary, comprehension, and total reading. 

Individual Teacher Characteristic Variables included 

57. Age. 
^ 38. Sex. 

39* Degree Status. 

^0. Total Years of Experience. 

• 41. Type of Appointment. 

^2* Teacher Expectancy of Student Performance xinder real 
conditions. 

^3* Teacher E^gpectancy of Student Performance xxnder ideal 
conditions. 

Ideal minus Real Teacher Expectancy. 
^5- Number of undergraduate courses related to reading. 
^6. Number of graduate courses related to reading. 

47. Number of inservice hours/month.- 

48. Minutes per week of teacher preparation for reading. 

kSm Minutes per week of teacher coordination time for reading. 
Student Body Characteristics of Reading Class Variables included 

50. Number of Students. 

51. Percentage of White Students. , 

52. Percentage of Black Students.. 

53* Percentage of Spanish Surnamed Students. 

Frequency of ^change xn reading group composition. 
55* Percentage Working Poor or Unemployed. 
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56. Percentage Unskilled >/orkers. 

57. Percentage Skilled Blue^ Collar. 
58» Percentage Skilled White Collar. 
59* Percentage Management Level. 
Go. Percentage Professional. 

61. Percentag^ V/orking Poor/Unemployed plus Unskilled Workers. 

G. School Characteristics Variables included 

62 Ability Grouping Practices 

H. Interactions of Input Variables included 

63. High Performing Students by Minutes Per Week (MPW), 

Whole Group Instruction (WGI), Small Group Instruction 
(SGI), Individual Help (IH), and Individualized 
Instruction (II) by the Teacher. 

6^*^ Low Performing Students by MPW, WGI, SGI, IH, ll by 
the Teacher. 

^ 65^ MPW Total Reading Instruction by Student Sex, Student 

Age, Number of Days Absent, Number of Pupils in Reading 
Class, High and Low Performing Students, and Teacher 
Ebqperience* ^ 

66* Student Sex by Teacher Sex. 

67. Teacher Age by Teacher Age. 

68. Teacher Experience by Teacher Experience. 

69. Instructional Materials by High Performing Studeats, 
Low Performing Students, and Teacher Preparation Time. 

OUTPUTS: . 

I. Student Readinfc Achievement on Criterion-^Referenced Tests included 

69-7if, Student raw score on ^th administration of CRT at Levels 
2, 3, ^1 5, 6, 7. 

J, Student Reading .Achievement on Norm-Referenced Test included 

75 • Student post test vocabulary Achievement Development 
Scale Score (ADSS) on June 197^ CAT. 

76. Student post test comprehension ADSS on June 197^* CAT. 

89 
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77* Student post test total reading ADSS on June 197^ CAT* 
Most of these variables were eventually included in a principal-compon- 
ents analysis. Thus, data reduction procedures, both logical and empirical, 
resulted in as few as 8 explanatory variati||^ in many of the multiple regres- 



si on analyses. " 
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4»0 Codlnf^ Scheme for Original Variables- 

In coding the original variables for computer analysis, provisions 
had to be made for continuous as well as categorical variables. The coding 
scheme that follows indicates the range allowed for continuous variables 
and the values assigned to each category of the categorical variables. 
For cases having missing values for particular variables, the mean for that 
variable was substituted. 



O 1 



O -15- 

ERIC 



Variables 



Code 



District Number: 



Utica 
Yonkers 
Syracuse 
Brentwood 



Year of Study 
School Number 
Student I.D. 
Reading Teacher Number 
minutes per week Reading Instruction 
Criterion Referenced Test Raw Scores 
Student Age in Half Years: 
Birthdate 



June 


1, 


1969 




Nov, 


30, 


1969 


Dec, 


1, 


1968 




May 


31, 


1969 


June 


1, 


1968 




Nov, 


30, 


1968 


Dec . 


1, 


1967 




May 


31, 


1968 


June 


1, 


1967 




Nov, 


30, 


1967 


Dec . 


1, 


1966 




May 


31, 


1967 


June 


1, 


1966 




Nov, 


30, 


1966 


Dec. 


1, 


1965 




May 


31, 


1966 


June 


1, 


1965 




Nov, 


30, 


1965 


Dec. 




1964 




May 


31, 


1965 


June 


1, 


1964 




Nov, 


30, 


1964 


Dec. 


I, 


1963 




May 


31, 


1964 


June 


1, 


1963 




Nov, 


30, 


1963 


Dec, 


1, 


. 19t2 




May 


31, 


1963 


June 


1, 


1962 




Nov. 


30, 


1962 


Dec, 


i. 


1961 




May ' 


31, 


1962 


June 


1, 


1961 




Nov. 


30, 


1961 


Dec , 


1, 


1960 




May 


31, 


1961 


June 


1, 


1960 




Nov, 


30, 


1960 


Dec, 


1, 


1959 




May 


31, 


1960 


June 


1, 


1959 




Nov, 


30, 


1959 


Dec , 


1, 


1958 




May 


31, 


1959 



33, 
44, 
55, 
88, 

00, 

00, 



99'. 
99. 



OOOaOO, - 999999. 
00, - 99, 
'000, - 999, 
000,0 - 999,9 



Student Sex: 



Male 
Female 



09.0 

10,0 

11,0 

12, 

13,0 

14,0 

15,0 

16,0 

17,0 

18,0 

19,0 

20,0 

21,0 ^ 

22,0 

23,0 

24,0 

25,0 

26,0 

27,0 

28,0 - 

29,0 

30.0 

01.0 
02.0 



Op 



Birth Order: First Bom 

Second Bom 
Third Bom 
Fourth Bom 
Fifth Bom 
Sixth Bom 
Seventh Bom 
Eighth Bom 
Ninth Bom Plus 

Father's Occupation: Working Poor 

( wel f a re/unenqj 1 oy ed) 

Unskilled Warker 
(gas station ^tteixdegitl. 

Skilled Worker-Blue Collar 
(fireman, electrician) 

Skilled Worker-White Colla 
(office worker, sales) 

Business 
(management) 

Professional 
(doctor, teacher) 

Father's Educationr None, or some grade school 

Completed grade school 

Some high school, but 
did not graduate 

Graduated from high school 

Vocational or business 
school after high school 

Some college, but less 
than 4 years 

Graduated from a 
4-year college 

Attended graduate ox 
professional achool 

Not available 



-17- 



Mother's Occupation| Working Poor 01,0 

(welf are/xineinployed) 

Unskilled Worker 02.0 
^ (gas station attendant) 

Skilled Worker-Blue Collar 03.0 
(fireman, electrician") 

Skilled Worker-White Collar 04.0, 
^ (office worker, sales) 

o 

Business 05iO 
(management , sal e s) 

Professional 0&.0 
(doctor, teacher) 



Housewife 07.0 " 

Mother's Education: None^.or some Rrade school 01.0 

Completed grade school 02.0 

Some high school, but . 03.0 
did not graduate 

Graduated from high school ^ 0 04.0^ 

Vocational or business ' 05.0 
school after high school 

Some college, but less than 06.0 
4 years 

Graduated from a 07.0 
4-year college 

Attended graduate or 08.0 
professional school 

Not available 09.0 
Pupil Evaluation Program (PEP) Reading Test Raw Score 00.0-99.9 
Number of Days Absent ' ' — 00.0 - 99.9 

Percent of Days Present ' , 00.0 - 99.9 



la- 
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classroom Ethnicity: Number of Students in 
^ ' Reading Class 



Number of White Students 

Number of Black Students 

Number of Oriental Students 

Number of Spanish Students 

Number of American Indian 
Students 



00.0 - 99.9 




Classroom tiES: 
(Occupation of head 
of household) 



Number of Vforking Poor 

Number of Unskilled Workers 

Number of Skilled Workers- 
Blue Collar 

.Number of Skilled Workers- 
White Collar 

Number of Business Management 

Number of Professionals 

flumber of Absentees in Classroom, per Day 

Number of Students Moving into Class during Year 

Number of Students Moving out of Class duriTig Year 

Educational Attainment Under Existing Conditions: 
"(Teacher Expectancy) 

8th grade or less 
9th, 10th, or 11th grade 
High School Graduate ' 
Jr. Collega, Business 
4 year college or more 

Educational Attainment Under Ideal Conditions: 
(Tedcher Expectancy) 

8th grade or less 
' 9^:h, 10th, Or 11th grade 
High School Graduate 
Jr. College, Business 
4 year college or moxe 



00.0 
00.0 
00.0 



00.0 
00.0 
00.0 
00.0 
00.0 



01.0 
02.0 
03.0 
04.0 
05.0 



01.0 
02.0 
03.0 
04.0 
05.0 



99.9 
99.9 
99.9 



00.0 - 99,9 



99.9 
99,9 
99,9 
99,9 
99,9- 
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Frequency of Change in Reading Group Composition: , 

No change Sept. - June 01.0 

Little change (every 5 rao. ) 02.0 ^ 

Some change (every 3-4 rao.) 03.0 

Much change (every 2 mo.) 04.0 

Number of Undergradualf«' Courses in Reading Instruction 00.0 - 99.9 

■ 

Number of Graduate Courses in Reading Instruction 00.0 - 99.9 

TOpw of Teacher Preparation for Reading Instruction 00.0 - 99.9 

mpw of Teacl^er Coordination of Reading Instruction 00.0 - 99.9 

Teacher Absences: Teacher not absent 2 01.0 

consecutive weeks 

Teacher absent for 2 02.0 
♦ consecutive weeks vv 

mpw of Non-instructional Reading Activities 000.0 - 999#r9 

Teacher Sex: Male 01*0 

Female 02.0 

Teacher Age: Year of Birth ' OO.P - 99.9 

Teadtjer Certification: None Ql.O 

5 yr. provisional 02.0 

♦ " 10 yr. provisional 03.0 

NYCS - Subst^^^E^ 04.0 

HYCS - Probaf/ionar^ 05.0 

Buffalo Toaporary 06.0 

Buf f albv^?robationary 07.0 

Perm^ent ^ 08.0 

Teacher Degree Statu:^r l^reshman year or less 01.0 

^ Soph, year completed 02.0 

Associate degree 03.0 

Junior year or less 04.0 

Bachelors degree 05^0 

' Bachelors degree + 30 hr. 06.0 

Masters degree , 07.0 

Masters degree + 30 hr. 08.0 

Doctorate 09.0 

Total Years Teaching iS^xperence 00.0 - 99.9 
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Type of Appointment: Civil Service 

/ Substitute 
Probationary 
Tenured 

Index of Material Resource Utilization 
California Achievement Test Reading ADSS Scores 



01,0 ' 
02,0 
03..0 
04,0 " 

00, - 99, ' 
000,0 - 999,9, 
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4,1 Coding Scheme of Trajisgenerated Variables 

In addition to the original vcu:iai)les already listed, the following 
new variables were created from theipriginal variables by trajisgenerations* 

Dummy Variables for District and Schools Each district and school 
building was assigned a \anique variable. Each student was then, assigned a 
value of 1 if in a particular district or school. He received a value of 
0 in all other dummy variables for district and»\school. 

Minutes per Year (ropy) of Reading Instruction. In an effort to use 
a more accurate estimate pf time spent in reading instruction for each 
student^ each of the time vciriables recorded in minutes per ,week was 
' transgenerated into minutes per year. This involved the number of sphool 
days per weejc, the average number of school days in a School year, 2Lnd the 
percentage of days present for each student • This transgeneration is 
staramarized in the following formula: 

MPY= ~i X 181.8 days x % days present. 

'5 ^ . 

Teacher's A^e. The teacher's age was determined by subtracting 'the 
teacher's year of birth from 74 (1974)^--^^ 

Teacher's Age= 74- date of birth. 
Percent of Whjte Stu(?ents in Reading Class. Percentages of reading 
class ethnicity (vfcj.te, black, etc.) and SES (working poor, unskilled worker 
etc.) were determined by dividing the number of students in each category 
by the total number of students in the reading class and multiplying by 100. 

4 * 

^*oi: example: * ^ 

. , • ^ , _ : - ntunbax o:lf/Hhi,t6' students 

% white students xn class = ^ ■ x 100 

total number of students 

Logarithmic Transformation . Natural logctrithms of all time variables 
" were also generated in order to make theJLr relationships in the regression 
models more linear. 98 



2- scores . While examining interaction effects, thes^ v^^able^ wefe 
standardized using 25-scores obtained by dividing the diffe-cence between the 
raw score and mean by its standard deviation or ; . ' 



ro Frequency Distributions of Major Original Variables 

The following histograms were plotted using a subsan^le of 2672 
students across all districts in the study^ The plots were made using 
raw data. . Intervals and ranges were determined by the computer program 
(BMD 05D). Variables having a high frequency of zeros or appearing to 
have low variability were plotted using another program (BMD 07D) which 
allowed for the Selection of special >^lues from the distribution and 
provided the exact frequenby. count for each interval. 
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6»0 Means and Standard Deviations of Major Variables by District 



The following table of means and standard deviations included thel 
major variables used in the study* The mean for each variable within 
each district is given with the standard deviation enclosed in 
parentheses. Values not a^vailable or not applicable to a given district 
Were so designated by a dash (-)• 
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7»0 Intercorrelatlons of Major Variables by District 



The Intercorrelations of the ^ariables used In the full model 
regression analysis for each district are presented in Tables 7.1 - 7.4, 
It should be noted that for df == 500, jc must be at least + .088 to be 
significant at the .05 level and + .11,5 at the .01 level. The variable 
nximbers correspond to the variables used in the regression .analysis for 
each respective district. Therefore the numbers are not consecutive. 
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8,0 Mult;ipl!^ Regression Analysis l ( 

fhe model Used in the multiple regression analysis the district and 

CRT subsamj^les was noted in the Main Report. \ 

■ ■ ' \ * . - . ' * ^ -% 

» The variables used and numbers identifying them are, listed for each 

district subsample in Table 8.1 and CRT Level 4 and Level 5 in Table 8.2. 

As noted in Table 8*1, the de^^endent variable, the CAT pQst*-t^st, was 

e^qpressed In Achievement Development Scale Scores (ADSS), \diile the CAT 

pretest ADSS v^^e further transformed^ to Zr-scores to facilitate their use in 
interaction^. The raw score was used for the 3rd grade Pupil Evaluation 
Program (P^) test of reading. The various modes of instructional time were 
expressed in the natural log of minutes per year. As with the CAT pre£<^testy 
the z-scores of the logs for the time .variables were used for interaction^. 

The same scores were used in the analysis of the CRT subsan5)les with 
one exception. The CAT pre-test was also expressed in ADSS form. In 
addition, a value of 400 was added to the raw score of the CRT Level 4, and 
' 500 to CRT Level 5. 

TaJ3ie 8^3 Jbs a, ?uramajT^ of the regression for the full model without 
interaction^ for the fpxar districts* Regression analyses for each district ^ 
both with and without interactions^ are presented ijfx Tai)le& 0.4'-8. 11. 

Likewise, Tables 8,12 and 8.13 summarize the regressions for the CRT sub-' 
samples for Levels 4 and 5. In comparing 3tudent performance on criterion- 
and norm-'referenced tests, 5 models were used for both levels. In the first 
three models at each level, the fourth test administra,tion of the criterion 
referenced test (CRT) is the outcome measure or dependent variable. The first 
model includes the first administration of the CRT as a pretest while the 
secoind model includes both, the CRT and the California Achievei^^ Test (CAT) 
as a pretest. The CAT is the pretest in the third model, ^^^^^ J^oii^th cind fifth 
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models include the ,CAT as the dependent 'variable while the CRT and CAT are 
the respective pretests. The complete regression analyses of the two sub- 
samples, are presented in Tables 8.14 through 8.23. 

' ■' i ■ ■ . - . 

Filially ^ five mora subsamples were analyzed using CRT Levels 2,3,4, 
and 5 and^Si^ CAT as outcome measures. These included theo CRT as pretest, 
time variables^ and student data. These regression analyses are presented 
in Table 8.24. 
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Table 8>r Key to Vari.able:s Used la.ThaFull md.el fiegye^j^jL9tx M^lySlg By Pl^strict 



Variable Name 

Student Achievement 

Post-test CAT Total Readlng-ADSS 



Student Characteristics 
Age in half yea;rs 
3rd Grade PEP Reading-Raw Score 
^{Te-test CAT Total Reading- z- Score 



District 



«8 



Teacher Characteristics 
Degree Status 
Age 

Reading Cla&& Structure * 
. Number of pupils 
7. TSJhite. Studetitis 
•/. Working Poor /Unskilled Worker 

Instructional Materials 
IMRU 



A 
7 



1 
3 
28 



5 
9 



4 
10 

37 



B 
7 



1 
3 
33 



5 
14. 



4 
15 
42 



C 
. 5 

1 
23 



3 
10 



2 
11 
13 



D 
7 



i 



5 

10 



4 
11 

38 



School Characteristics 



Duniniy 
Dummy 
Duimny 
Dunmy 
Duimny 
Dummy 



Variable 
Variable 
Variable 
Variable 
Variable 
Varlabl e 



for School 
for School 
for School 
for School 
for School 
for School 



Instructional Time (minutes per year) 
Log mpy Whole Group teacher 
Log mpy Small Group Teacher 
Log mpy Individual Help Teacher 
LofS mpy Itidlv^Lduallzed Instruc. Teacher 
Log mpy Tot;al Specialist Instruction 
Log mpy Total Paid Aide Instruction , 
Log mpy Total Unpaid Aide Instruction 
Log mpy Total Reading Instruction 

Interactions of z-score Pretest CAf x Time 
z-score log mpy Whole Group 
z-score log mpy Small Group 
z-score log mpy Individual Help 
z-score l(^g mpy Individualized Instruc. 



17 
19 



27 



33 
34 
35 



8 
9 
10 
11 
12 
13 



22 
23 
24 
25 

30 
31 
32 



38 
39 
40 
41 



6 
7 
8 
9 



8 
9 



14 
15 
16 
17 

19 
20 
21 
23 



28 
29 
30 
31 



18 
19 
20 
21 

26 
27 
28 



34 
35 
36 
37 



ERIC 
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Table 8.2 Key To Variables Used In Full Model Regression Aipilysls of CRT Subsamples 



Variable Names - 

Student Achievement 

CRT Test Administration ^Raw Score 
CAT Post-test Reading Total-ADSS 

Student Characteristics ' - 
CRT Test Administration 1-Raw Score 
Age • 
Father's Occupation 
3rd Grade PEP Raw Score 
CAT Pretest Reading Total-ADSS 

Teacher Characteristics 
Degree Status 

Age ^- . . 

Re&dlng Class Structure 
Niii^ber of Pupils 
% White Pupils 
% Working Poor 
% Unskilled Worker 
7. Skilled Blue Ccdlar Worker 
7. Skilled White Collar Worker 
% Business ^ 
% Professional 

Instnictlonal Materials 
IMRU 

Instructional Time (minute^ per year) 
Log mpy Whole Group Teacher 
Log mpy Small Group Teacher 
Log mpy Individual Help Teacher 
Log mpy Individualized Instruction Teacher 
Log mpy total Specialist " 
Log mpy Total Paid Aide 
Log mpy Total Unpaid Aide 



Level 4 



2 
10 



1 
3 
4 
5 
9 



7 
11 



6 
1^ 
13 
14 
15 
16 
17 
18 

8 



19 
20 
21 

22. 
23 
24 
25 



CRT Subsamples 



Level 5 



2 
10 



1 

3 
4 
5 
9 



7 
11 



6 
12 
13 
14 
15 
16 
17 
18 

8 



19 

20 
21 
22 
23 
24 
25 
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Table 8.3 Raw Regression Estimates of Rfeading. Program Components 

o^k atudeat Performance. Vti±Iiixi-.SchQQl ^;^i^t;^i;pt3 



District 



Explan atory Variable 

Student Characterrstics 

CAT Ability (PrelfeGt) 
P2P Ability 
Age 



A {N:^567) B (N=947) C (N=967) D (N=523) 



50.032*** 53,636*** 48.932*** . 52*662**^ 



.509** 
1.049 



.811*** 
.897 



.892 



.615*** 
1.077 



Teacher Characteristics 
Age 

Degree Status 

Reading Class Structure 

.Number of Pupils 
% IVhite 

% V7orking PoorA^n ski lied 
Worker 



.061 
3.051 



.099 
,729*** 

,160 



.066 
-.342 



.127 
.040 

.130 



-.25^ 
-.317 



.382 

.562*** 

.098* 



.472* 
2.292 



.751 
.057 



• 081 



Instructional Materials 
IMRU 



,157 



.233 



.447 



.020 



School Chatacteristics 

School A 
- School B 
School C 
School D 
School E 
, School F 



-1.923 



-4.560 
-.037 
8.606 
6.510 
6.184 
8.731 



26.559*** 

8.955 
28.521*** 
23.432* 



-21.741 
-.571 



Instructional Time ^ 

Teacher Whole Group 
Teacher Small Group 
Teacher Individual Help 
Teacher Individualized 

Instruction 
Total Specialist 
Total Paid Aide 
Total Unpaid Aide 



.637 
.289 
.040 



2.329 



.435 
.655 
-1.712 

.254 
-1.106* 
-2.583*** 
-2.205 



1.125* 

.511 
4.803*** 
-1.010 

-1.793* 
-1.231* 
- .203 



-.818 
-.435 
.256 

-1.746 
-3.084*** 

.075 
-6.941** 



ERIC 



Multiple R 
Multiple 
F Vci'lue 



.880 
' ' .774 
135.353 



.893 
.797 
164.457 



.870 
.757 
154.841 



.899 
.808 
117.657 



Note; ^ dash (-) in this table indicates that the variable does not apply in that 
di.strj.ct. , The dependent variable is the Calif oniia Achievcinent Tcst^ 

a 

All tune varicibles are natural log transformation of minutes per year. 
*E.<'«05. **E^<.01. ***p<.001. 
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Table 8,12 *.Raw Regressipn Estimates of Criterion and >Norm-Ref erenced 
Pferformante in^'tull Explanatory Models 
CRT Level -4 Sub-sample (N=607) 



Outcome Measure 



pretest 

. ^riterion-Referenced ^Test 
California Achievement 

Classroom Conditions 

No. of Pupils 
% VThite ^ 
% Working Poor 
% Unskilled Worker 
% Skilled. Blue Collar 
% Skilled vniite Collar 
% Business . ' 

Professional 

Teepher Back grpund 
Age 

Degree Status 

Student Aptittide 

P\ipil Evaluation Program Test 

Student Background 
Age 

Father's Occupation 
Quantity of Instruction " ^ 



1 


2 


3 


4' 


5 


CRT 


CRT 


CRT 


. CAT •* 


CAT 






\ 
















.426*** 


,309***' 


D 


3.103*** 


n 

LJ 


D 


.042*** 


.053***^ 






-.173*** 


-tl50*** 


-.180*** 


. -.011 


• .1(66 ■ 


^ .053**]^ 


.029*** 


.037*** 


.481*** 


.185** 


♦ -.023* 


-.016 


-.-023* ^ 


-.157 


-.104 


-.008 


-.002 


-.004 


-.133 


-.052 


-.039*** 


-.033*** , 


-.048*** 


■ 00.8 


V.005 


-.018 


-.022 


-.027* 


-.034 


-.111 


-.001 


.003 


.019 


„ -.422* " 


-.290 


.062* 


.081*** 


.089*** 


.405 


.714*** 



.037* 
.544** 



.129*** 



.006 
.253* 



nipy Whole Group Teacher -.170* 

mpy Small Group Tea,cher -.084 

mpy Individual , Help. Teacher .073 

lopy Individualized instr. Teacher .167 

©py Total Specialist ,027 

mpy Total Paid Aide ..013 

tal Unpaid Aide -.162 



,035* 
-.626*** 



.156** 

.011 

.087 

.097 

.013 

.082. 

.089 



.042* 
.755*** 



.063**- .075*** 



,098 -.109 
,281** ^ -.291** 



-.191** 

.004 
-.001 

.069 

.055 

.063 
-.007 



.077 
.144 



1.533*** 



2.825** 
2.071** 



.491 
-1.045 
-2.030 
-.199 

.027 
-2.325 
-2.811 



.079 
-1.703 



.673*** 



1.326 
1.636* 



' .521 
.039 
-2.263* 
-1.299 
.036 
-1.456 
-1.410 



-''Tkiltiple R 
SyJIliiltipie R 
F Value 



.66(5 
.470 
23.495 



.747 
.558 
31.977 



.705 
.497 
26.172 



.677 
.458 
22.453 



.804 
.647 
48.610 



ERIC 



Kote: "D" ifidicatea variable -was deleted from regression model. 

** £ ^ .01.. , . 181 ■ 

*** £ 4. -001 
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Table 8»13 Raw Regression Estimates of Criterion-and Norm-Ref e^renced 
Performance in Full Explanatory Models 
CRT Level 5 Subsample (N=497 ) 



" • • • / 

Ex.- ".2natO¥3ir VariabJ.gs . / 



Pretest 



Outcome Measure 



1 

CRT 



Criterion-Referenced Test ^.556*** 
Calif dxmia Achievement D 

Classroom Conditions 

NO. of Pupils -.177*** 
' % Vftiite .022 

% Working Poor -.013 

% Unskilled Worker -.024* 

% Sillied Blue Collar • -.054*** 

% Skilled \^hite Collar -.010 
% Business ' .,048 
% Professional .004 



.431*** 
.038*** 



-.224*** 

-.oil ' 

-.007 
-.025* 
-.052*** 
-.018 

. 062* 

.006 




D 

.061*** 



.385*** . 

.004 

.001 

.041*** 

.080**^ 

.028 

.091** 

.012 



3.450***- 
D 



"27494*** 
. .409*** 

.047 

.288** 
-.037 

.196 
-.581* 

.706** 



5 



B 

.758*** 



1.193*** 
.192* 
.182 
.229* 

-.070 
.022 

-^252 
.691** 



Teacher Background 
Age 

Degree Status 



.029 
-.389 



.017 
-.455 



,01:7 
.503* 



.\498* 
.468 



.265 
-.976 



Student Aptitude 

Pupil Evaluation Program 



.043 



-.008 



-.012 



1.273*** 



••319 



Student Background 
Agp 

Father's Occupation 
Quantity of Instruction 



-.062 

^r024 



-.168 
-.043 



.312* 
.070 



-1.948 
.733 



ERIC 



Note: "D" indicates variable was deletied from regression model. 
' *E<»05* **£<.01 ■ ***£<. 001 

182 
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-4.281*** 
' -.565 



mpy Whole Group Teacher 


-.242** 


-.168* 


-.188* 


-1.018 


.285 


mpy Small Group Teacher 


-.395*f* 


-.380*** 


-.577***' 


.692 


.453 


mpy Individual Help Teacher 


.634*** 


.559** 


.448** 


2.554 


.998' 


mpy Individualized Instr. 












Teacher 


.078 


.212 


..360* 


-3.063* 


-.183 


mpy Total Specialist 


-.110 


-.03^ 


-.130 


-2.53 


-1.347 


mpy Total Paid Aide 


-.195 


-.088 


. 047 


-.699 


1.638 


mpy Total Unpaid Aide 


48.889 


57.424 


" 94.200 


-237.244 


17.895 


Multiple P.^ 


.758 


.783 


. 730 " 


,632 


.776 


MujLtiple R 


.57$ 


.613 


.533 


,399 


.603 


F Value 


30.619 


34.154 


25.804 


15.026 


34.285 
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9,0 Studies of CiayylJLj;^ea,rj,ty a^d CuJ^yULig^e^r Ixlt enaction. 

Due to tha coinplexity of the time effects experieaced in. the Piain 
regression analyses, detailed curvilinear and interaction studies of specific 
time variables were initiated* 

Curvilineax time studies were designed to explore the linear and qua- 
dratic effects of each time Variable with the California Achievement Test* 

The procedixre involved successive multiple regression analyses in the follow*- 

ing order: ^ 

J» Regression of Achievement on time OLinear effectl, 
2* Regression of achievement on time and time squaored Cquadratic 
effect) ^ 

3* Regression of achievement on the full explanatory model variables 

including time Clinearl, ' 
4» Regression of achievement on the full explanatory model variables \^ 

including time and time squared (qaadratic), 
A key to the variables used in these regressions is included in Table 
9,1. 

The -presence of -a curvilinear effect was determined by testing the 
significance of the P-^ratio between step 1 and step 2. ' A significajit P-ratio 
between the linear and quadratic (components indicated a curvilinear ^ffect. 
In cases where there was no significant curvilinear effect, the f^ratio 
between zero and the linear component was tested for significance. A sig- 
nificant 1Fi-ratio indicated a linear effect, while a non^jj^ignificant P^ratio 
indicated no effect of that time variable on achievement. 

For significant effects, a regression equation "Was solved and plotted us- 
irig 'raw regression coefficients. A clarification of the stability of the slgnlfl- 
aant time effects was provided in step 3 and 4 *when each effect was included in the 
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full e^qplanatory model. The regression analyses of t^fe significant 
variables (alone and within the full explanatory mod^l) and the plots of 
their regression equations are included in Tables 9.2 tftifough 9.21 and Figures 
Figures 9.1 through 9.10. / v - 

The regression equation, uged to plot tlife sigtLif^cartt cviryilinear time / * 
effects wa,^ of the geaeral foxtair ' ' 

y ^ ^0 + a^X + a^X^ 

where 

Y = Calif omia^Achievement Test (ADSS), ^ 
X 5= 4:ii9tpiction^l time in itiiautes per ]^eax X ^Ql, 

<^_jf ^iater<^pt,. 
a^^ and a^^ = r^V regression coefficients- 
It was determined that ^instructional time might have a different ef feci: 
on students )gith d^ffer.ing ability. Due to the possible curvilinear effect 
of tinje^ a model for studying curvilinear interaction was requixed^ This 
model took the following form;^ 

(Model 1) y + a^X^ + a^X^ + a3X3 + a^X^ f a^X^ + a^X^ + 

where ^ 

Y = California Achievement Test (ADSS), 

^ 1 it the student was of high ability (scored above 467 on the * 
. pretest of the CAT) ; otherwise X^^ = 0, . 

= 1 if the studenj:-^as of low ability (scored below 368 on the 
pretest of the CAT); otherwise 0, 

X^ = instructional time X .01 for high ability students; otherwise 

X3 = 0, 

X^ = instructional time X *01 for low ability students; otherwise 
X- = squared elements in X 



.This model and subsequent testing procedure were derived from the same 
model suggested by F. Kelly, D. Beggs, K. McNeil, T. Eichelberger , and 
J. Lyon in Multiple Regression Approach, Southern Illinois University 
Press, Carbondale, 1969, pp. 173-191. 1 Q*^ 
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- thja dif Jfexeace between, thja observed Y and pJTedicted^ y, 
^0 " y^in-texcept, . , 

^1' ^2' ^3' ^4' ^5' regression, coefficients^ 

This model euid reduced forms thereof provided a mechanism for study* 
ing the curvilinear interaction of instructional time by sttident pretest 
ability when using the California Achievement Test as the dependent variable. 
The mechanism involved a series o:f^ tests based on change in R from model to 
model. These models and tests wiere iJerfoxmed in the following steps; 
Step 1; Model 1 was tested to 's^e if R was significantly different from 

, ■ -.If : 

2 

, zero. If R was signififetot, the possiblity of curvilinear inter- 

V 

action existed in the data. 
Step 2 ; To deteuoine whether the relationship between the instructional 

^^r;^.tiine^ variable and reading achievement was linear and different • 

2 ' 2 
slcross ability, the R • of model 1 was compared with the R of the 

following Model 2: 

(Model 2) J er^ + a^X^ + a^X^ + a3X3 + a^X^ + ^ 

Xt can be seen that^odel 2 is actually Model 1 without the squared 
terms for instructional fllme within an ability group. A non -significant F- 
ratio between Model 1 a^d Model 2 indicated tha?t the lines were linear which 
led directly to further tests in Steps 3 and 4 to determine the nature 
of the linear lines, m significant. P-ratio between Model 1 and Model 2 
indicated cxmvilinearfity which required further testing' for interaction or 
parallelisn/in Steps^ 5 and 6. ' ^9 
Step 3 ; If the linj&s were determined in step 2 to be, linear^ the R of Model 

2 (which dillows the lines^to be non -parallel) , was compared with the 

. 2 J ^ ' ^ 

R of MOc^el 3 (which allows the lines to be parallel): 

196 
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1 3 4» 

If the F^ratio between Moded 2 and Model 3 was sigiudfteaat , the non^- 
paraJLlel lines were plotted xising Model 2^ A 3ign£fifcant F^roratio ruled out 
non-paralleJ. lines ^ leaving the possibtXi^ty of parallel or superimposed lines 
to be tested £n Step 4, ^ 
Stej2_4: Having ruled out nonr^parallel lines, Model 3 vas compared with Model 
4; ^ 



CModel 4) ° + a^Z^ + E 



'4* 



A significant F-ratio between Model 3 and Model M indicated that the 
lines were parallel and Model 3 was plotted. An non-^sVgntficant P-ratio 
indicated that the lines for each ability group were yuperisiposed and one 
line plotted by Model 4 best described the data. 

' Step 5t As stated^ a significant F-ratio between Model 1 and Model 2 indica- 
ted cunyfed lines* In this case. Model 1 was then compared to Model 5: 
CModel 5) ^ a^ + a^X^ + a^X^ + a^Z^ + a3Z3 4- 

where Z = + X. and Z * « X^ + X,. 

2 3 4 3 5 6« " ^ 

A significant F-ratio between Model 1 and Model 5 indicateifStirvilinear 

interaction, and this interaction was plotted using Mgdel 1. If, on the other 

hand^ the F-ratio proved non^significant, the ciarved line's were either equi- 

distant "(parallel) or superimposed. ^ 

Step 6 ; Finally, .Model 5 was compared with Model 6: 

^del 6) « a^ + a^Z, + a^Z^ + E^ 
J. U z 1 o 2 6* 

If the P-ratio between Model 5 and Model 6 was stgnific^iit^two parallel 

curved lines were plotted using Model 5. If it was not significant, one 

cxnrved line was the best fit and was plotted using Model 6. 
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A regressJLoa aae^lysis ba.se<3 oa ttys-^e ^Ix ;nodeJL^ wa.s cojpguted orv six 
different ioistructioaal time y^rtajDles J3x aXX fovp: 9c1xqo1 distjctcts* The 
variables used in each model are smnmari^ed Irv Table 9» 22# The regression 
analysis of the Jtiodel of best fit for each time variable ia ea,ch district is 
iacluded together with ii:s plot in Table 9# 23 through 9,45 and Figures 9»11 
through 9.33# 

' ' • 
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Table 9#1 Key to Variables Used in Curvilinear Regression 
Analyses by District 

Variable Name District 



Student Achievement 

Post-test CAT Total Reading-ADSS 

Student Characteristics 

Age in half-years \ 
3rd Grade PEP Reading-Raw Score 
Pretest CAT Total Reading-ADSS 

Teacher Characteristics 
'Degree Status 

Age , ^ ^ 

Reading Class Structure 
Number of Pupils 
7. White Students 
% Working Poor/dnskilled Worker 

Instructional Materials 
IMRU 

\. 

School CharacteristfLcs 

^ Dummy Variable for School 1 

Dumtty Variable for School 2 

j Dimmy Variable for School 3 

Dummy Variable for School 4 

Dummy Variable for School 5 

^ Dummy ^'ariable for School 6 

Xnscructional Time (minutes per year x lOl) 
Total Individualized Instruction . 
Total Teacher Instructional Time 
Total Specialist Instructional Time 
Total Individualized Instruction 

Time Squared 
Total Tedcher Instructional Time 

Squared ^ . " 

Total Specialist Time Squared 
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Table 9*22 Variables Used in Models 1 through 6 for Regression Analyses 

of Curvilinear Interaction 



\ 

V 



Variable todel 
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— i — 
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Cfl.TWVDSS 
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D 
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= 1 if high, abi-lity 1 
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' X 




X 
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= 1 if low ability 
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X 


X 
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5= time of high ability 


X ,01 




X 






J 
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« time of low ability X ,01 


X 


X 
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X3 =4 




X 












10 


















11 


h " ^3 ^4 .! 
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X 
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12 


2 2 I 
^2°^3'*-^4 ' 












X 


X 



Note: Variables 2, '4, and 5 have been omitted since they, were used only to 

trl|isgenerate^ variables included in the models 

• • ^ 

^"D" indicates dependent variable** 
"X" indicates variable included in model. 
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PREFACE 

To conduct 9, re^ecgcclt 5t;udy in. public, educa^tipi?. requires tHe coopera,t4:oa 
aivd-e^ffo^pt of' a, la^^B xvaiqber of people. Especially importarvt a.xe th& a,tti1;ude 
and willingness of school adiniaistxa,tojr^. to support and contribute to xesearcli 
efforts tlxat generally tAtie and resources^ beyond normal operations. The 

contriJmtion of people in th& Seneca Central School District to the study 

as reported is woXthy of commendation, • ' ■, 4 

There were so many people, teachers,, clerical personnel and especially the 
students themselves of the West "Seneca District \dxo contributed to the research 
effort that to single out a few seems arbitrary. Never'theless, I must hazard 
that risk and mention a few individuals without whose cooperation and effort 
study could not have taken plack, Fixst, Dr, Carl Markello, Deputy Super- 



iitendent of Schools, whose leadership over the three years was extremely sxip- 

portive. Rarely does one fiixd a school administrator with, the knowle'dge and 

passion for research of Dr. Markello, Both the principal and assistant princi- 

/ 

pal of the Northwood Elementary School (Wr^ Al Wolchuck and Mr. Steven Maricich, 
respectively) cooperated and supported the study in innumerable ways. Two of 
' the finest teachers I have had the occasion to work with, Ms. Mary Alice i^alz 
and Ms. Ann Wojiechowich, demonstrated that quality of dedication to the pro- 
fession that is often overlooked these days. Without their hard work and 
patience with the researcher, the study could not have been completed, finally., 
a special commendation for the skill and patience of Ms. Doretta Dodgq for her\3 
assistance in the tedious task of data tabulation and analysis^ 

All of these, both mentioned by name and those un-named, have my sincere / 
thanks for all the effort expended over tte pa,st three years. 
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I . Introduction 

— • ~— -V ^ 

*ln the school year 1971-72 the Erio County Dt^ard of Cooporatlvo Educational 

Services #1 (BOCES) offered a new criterion referenced curriculum developrrent 

and evaluation service to local school districts on a pilot' basis. This service 

Is ca-lled "the System for Pupil and Prqgram Evaluation and Development (SPBED) 

which is a dfevelopmental project- sponsored by the Bureau of School and Cultural 

Research of the New York State Education Department. The primary component of the 

SPPED that was implemented on a pilot basis was the Comprehensive Achievement 

f 

Monitoring (CAM) evaluation system. 

CAM was originally initiated under a grant from the Char(e3 F. Kettering 
Foundation to Dwight W. Allen at Stanford University in 1967. Primary ^eve lope r 5 
of the system v/ere William P. Gorth and Paul Pinsky. In 1968 the project moved 
to the University of Massachusetts where it came to fruition. CAM has undergone 
further refinements in New York State under the direction of Robert O^Rel I ly. 
Chief, Bureau of School and Cultural Research. 

CbmprehensI ve Achievement Monitoring is a computer supported pupil and program 
evaluation system based upon a criterion referenced model of evaluation. CAM 
focuses on students' achievement of specific learning outcomes (b^haviora I ly stated ^ 
oBJectives) rather than on global educational outcomes. CAM requires the 
specification of a complete set of objectives for a course and development of 
specific criterion referenced test items to rreasure each objective. Thus, CAM 
focuses on the actual achievement of specific behavioral ot^Jectives rather than " 
relative <or norm-referenced) measures of achievement. ^ 

The evaluation design underlying the CAM approach is quite dlffeiN^nt from 
tfie more traditional approach to student and program evaluation. Each course 
objective is tested frequently throughout the duration of the course (semester or 
school year). 
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At each test admlh is^trat ion, porformanco on objectives not 
yef taught i-s proti^slod, performance on objectives just 
taught is immediately post-test^icj, ahd por^f ormonce on ob- 
jectives laught earlier in the cQurso is measured for 
retention. Parallel test forms^ t:omparab le in difficulty 
and content, are all used at each test administration, but 
each student receives a particular (test) form only . once 
during the course. Each form typically has an item for 
each objective. Each item is used on only one test form. 
Thefunction of a particular item changes in relation to 
the time at which the objective it is measuring is taught. ^ 
Testing may take place at regular intervals (e.g.^ every 
two weel<s) or at the end of certaJn instructional units, 

Due to the frequency of testing, the large amounts of data analyzed, and 
the numerous type^ of decision-relevant reports possible, computerized data 
processing is tjie on ly. ef f ic lent and effective system to employ. Output from 
CAM Computer programs at each test administration provides the fo I lowi^ng^ in- 
formation: 

1. Individual Student Data - a total score for that test and each 
previous one. A total score on the objectives for wtiic'h the / 
student has received instruction for that test and each previous 
one. A correct-incorrect indication for each item on the test 
coded to the specific course objective/ it measures. 

2. Group Summary Data - for each group or subgroup of students * 
(e.g., class, ability group, grade level, etc.); the percentage 
correct from all test items on all forms for each objective or' 
groupings of- objectives such as modules or units. 

3. Item Analysis D^ta - period ica I ly as desi red,., but usually at the 
end of a course, data on each item is provJded. The Information 
treats each item by its three functions, pretest, posttest, >e- 
tention and provides data on its difficulty level and the dis- 
tribution of choices for multiple choice distractors. 



1 ' 

Gorth, V/illiam P. "Comprehensive Achievement Monitoring (CAM): A Project to 

Develop Longitudinal Classroom Evaluation Using Item Sampling."^ Paper 

presented at National Council of "Measurement of Education. New York, 

February, 1971, p. 7. ' ^ 
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Comprchonslve Achlcvoment Nbnitoring, then, is a syslemotrc way of measuring 
'student achlevemont of specific learning outcomes employing a longitudinal test- 
ing design and the use of computer programs for rapid data analysis and reporting 
of the results to students, teachers^and others. In addition to providing 
specific information on Individual students, the summary data Is useful for pro- 
gram eva I uation.^ 
I I • Research Problem " 

During the first year In which the BOCES offered the CAM evaluation com- 
ponent of SPPED to a few selected school districts, one of the pilot districts. West 
Seneca, agreed to conduct a research study cooperatively with the ^ES, Both the 
local school district and the BXES Instructional Services staff v/ere^ Interested 
In determining the impact this technological innovation would have upon the school 
program and student achievement. The local school district personnel were primarily 
concerned with whether or not the utilization of CAM would improve student achieve- 
ment. Both the pilot district and BOCES were hopeful that student achievement in 
reading v/ould be positively affected by employing this computer based evaluation 
system. The basic premise underlying the employment of the CAM system was that the 
receipt of CAM test results on a regular basis would provide students and teachers- 
with relevant data at appropriate times that v/ould Increase the effectiveness of 
the teaching- 1 earning process. The BXES researcher was also Interested In CAM's 
Impact upon student achievement but also wished to ansv/er the question, "Which 
form of program' eva I uat I on, Norm-referenced (NRT) or Criterion-referenced, (CRT) , 
Is a more sensitive measure of student learning outcomes?". 

2 ~~ * ^ 

For a more detal led expl anatlon of the CAM system see: 

Gorth^ .Wl I Mam, O'Reilly, Robert and Plnsky, Paul, Comprehensive Achievement 
Men I torinq, Amherst, Massachusetts; University of ^ Massachusetts , 1974. ^ 
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Tne pilot school district and the DOCES rof:.ocjrchcr c)(jrced that '\he proposdtl study 

wcjic^ address Itself to both concerns. 

Evaluation specialists as well as Pvubllc school personnel hovo been examining 

and debating the CRT vs NRT question for a number of years. ^ The general con- 

teniion of adherents of CRT. Is that since "A cri terion-rof erenced test is one' that 

Is deliberately constructed to yield measuroments that are directly Interpretable 

4 

In terms of specified performance standards", they are more appropriate measures 
of the Intended outcomes of an instructional program. The use of CAM Criterion 
referenced tests as well as a standardized norm referenced measure to assess pro- 
gram outcomes In the pilot school district provided an opportunity to I nves'f'l gate 
this contention. 

i * 

Thus, there were two basic propositions under Investigation:' 

1, The use of the CAM eyal uatlon system v/III significantly Increase 
student achievement. • • 

2. ' Criterion referenced measures used In the CAM systojn are moro 

sensitive In detecting Instructional program outcomes than- 
standardized norm-referenced measures. ^ CT • 

Reformulated Into research hypothesis: 

H.: There Is no -si gnl f leant difference In the Increase In^stud^nt 
achievement between those studenis Involved In a program using 
the CAM system than those students not using a CAM system.- 

H : There Is no significant difference In the achievement levels of 

students as measured by e^lther criterion referenced tests (CAM) ^ 
or standardised norm referenced tests (Metropol Itan Achievement 
' Test). 
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See Cronlack^ L.J« "Course Improvement Through Evaluation" ' 
Teachers Col logo Record, 1963, 43, 672-683 and *: , 

Barnabel, Raymond afid Leies, Sam. Dehavl.Qral Objectives In Curriculum and 
Eya I uatlon . Dufuque: Kondall/Hunt Publishing Company. 
Rpss, Paul C. "Some Considerations In the Design and Use^of Criterion-' 
Referenced Tests". Paper presented at Northeast Educational Research 
Association, March, 1970. 

Robert Glazor>and Anthony J. Mitko, "Measurement in Learning", In Fducatl ona I 
I'teasuremen t/T oditod by Robert L. Thorndike, WashIr\gton, D.C^ American Cou,ncI I 
ofK£>tiCatWn , 1971, p. 653. ' \ 
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I I L Research Doslqn and Mothodolfrgy 

In conducting rosoarch studios in Education the operational circumstances 
of public schools /often preclude the establishment of rigorous research designs 
prior to undertaking .a spec i f ic investigation'. Such v/as the case in this study. 
Though the local school district, West Soneca, v/as quite cooperative throughout 
the entire study, much of the data for the study was based upon that which was 
available via normal school 'operations. Also, much of the design could be considered 
post ^tioc formulation since the speei f icaf ion of the design evolved, over the three 
year period of the study. 

At the commencement of the study, approximately December of 1971, the West * 
Seneca Central School District had been utilizing the CAM evaluation system in, the 
Intermediate reading program (grades 4r-6) in one elerfentary school buildtng since 
the beginning of the 1 97 1 -72Hchoo h year. During the previous summer five teachers 
fron the school district attended a four week workshop where they developed the 
instructlonaf objectives and criterion referenced test items for their reading 
program. Therefore, the program had been In operation for a few months prior to 
the decision to conduct the study. ^ , 

The Initial research design of the study was an "exper Imenta l-control" 
design employing pre-and post assessment of student achievement In reading. 
The school district selected two elementary schools, the Northwood and Clinton 
Elementary Schools, to serve as the- sample popu lati ons- for the study. Both 
schools serve a student population residing In the same geographical area. The 
district was of the opinion that the background of the students. In terms of 
socio-economic status, wealth, ethnic origins, etc. was very similar for both 
school- populations. The Northwood School which was using the CAM system was 
the experlrrental group and the Clinton School was the control group. 



In this firsi year of tij<^s+udy (the extension pf the study into a three 
year iongitudinal design was ddifepmined after the resuits were reviewed at the 
end of year one) two groups of students in>ach schooi were seiect^_d as the 
subjects for the study. All thj^Hourth and sixth grade students receiving 
regular instruction in reading were chosen to provide the foi lowing sample 
populations: 



Experimental Control Total 

(Northwood) (Clinton) Students 



4TH Grade 15? / 139 293 

6TH Grade 131 89 210 



SInfce the effect of CAM on student achievement in the reading program was 
the major variable under study, controls for as many other variables as possible 
were attempted. -The characteristics of the teachers in both schools (age, 
experience and " trai n ir^ were examined. The teaching populations In both sthools 
In terms of these variables were very similar. The currlcutum anrf Instructional 
resources (basai readers) were the same for both schools. The policy on student 
grouping for' i nstructlon wap the same for. both schools. The only major difference 
between the two groups wa^ that some of the teachers in the experimental group 
received four weeks of training during a summer workshop on establishing and 
uti Uzing a CAM evaluation system. 

Probably the most Important variable for which controls were needed was 
the achievement level In reading of both student groups at the beginning of the 
school term. Shortly afte.r the beginning of tlie 1971-72 school year, ¥he standard- 
ized Metropolitan Achieverrrant Tests (MAT) were admlijlTstered to both groups of , 
students. The resuits of the reading section of theoe tests are reported in 
Tables i'and 2. 
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TABLE 1 



Comparison of Moan Standard Scores on the Pro-f4AT - GracJo A 



Moan Standard Scoro 
Standard Deviation 
Sample Population Size 


Experlmenta 1 
Group 


Con+ro 1 
Group 


65-9 
11.6 
156 


63.0 

13. I. 
152 


TABLE 2 

Comparison of f^an Standard Scores on the Pre-MAT - Grade 6 


Mean Standard Score 
, Standard Deviation 
Sample Population Size ' 


Exper 1 menta 1 
Group 


Contro 1 
Group 


83.4 
16. 1 

133 

- 


84.3 ' 
12.7 
98.- 



^ The experimental group at the fourth grade level has a statistically sign- 
ificant higher mean score on the pre-j^-est than the controj group. The calculated 
statistic was 2.0567 which was significant at the P<^^05 levfel of error. At the 
sixth grade level the reading ability of the students, as measured by th© MAT, wa^ 
almost the same, there being only ,9 dif.ference In the mean standard scores of 
the 'hvo groups. A T test Indicated no significant dlfferenqe. Thus, the sixth 
graders In the two groups began the school year at approximately the same achle\/e- 
ment level^, but the fourth graders In the exper I rr.enta I group begary"the year with 
a higher achievement level than thef control group. Since the school year was well 
-underway when the research sj fudy J xSgan, It was fiot possible to obtain pre Instruct 
Ion measures with the CAM tests for 1ho ccrn"trol group. 



28^ 

-208- 



Teachers from thA exporimental group developed a set of reading objectives 
during .a summer training session. In addition fo a set of Instructional ob- 
jectives for'^both fourth and sixth grade reading courses the teachers also de- 
veloped a set of criterion referenced test items to measure object i ves • At 
the begirtning of the school year the fourth and sixth grade teachers from the 
control group were presented with the set of reading objectives and were asked 
to Indicate which objectives they planned to utilize in their reading program. In 
all cases, the teachers of the control group indicated that over 95^ of the 
objectives were ones that they attempt to teach toward in their program. Thus, 
both the experimental and control groups were similar (with the exception of 
higher achievement level at the 4th grade by the experimental group) "and both were 
being taught the same basic reading program with the same instructional^ resources. 
The major difference between the two groups was the utilization of a CAM evaluation 
system by the experimental group. Eight times a year the experimental student 
' groups v/ere given CAM criterion referenced tests and the students and their 
teachers received analysis reports on the results. 

The final (eighth) CAM test was administered at the end of the school year ^ 
to both the experimental and control groups of students. The scores on these 
tests were used as the post instruction criterion referenced measure of student 
achievement In reading, as were the scores on the norm referenced standardized 
Metropolitan Achievement Tests in Reading which were als6 administered in June 
of 1972. Mean scores for both tesis for each group were computed and tests were 
employed to determine if there were'any significant differences in student 
achievement. 

■V 
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Af+or a roviow of the first year findings by the school district administra- 
tion ar.d tho BOCLS resoarchor, it was decided that the study should be continued 
for at least another year. 

<• Due to organizational arrangements and the procedure for the assignment of 
students to classes, ail of the students who^ parti ci pated in the first year of 
the project were not involved in the second year. The sixth grade students in 
year ona*were dispersed into middle school programs which were not using CAM,, 
thus it was impossible to follow their progress In the second year. The fifth 
grade population in both the experimental (CAM) and control groups (fourth 
graders the previous year) were the populations under study In the second year, 
Table 5 depicts the populations involved in this study. 

TABLE 3 



• — — ■ 

5th Grade 


Exper i menta 1 
XNorthv/ood) 


Control 
(CI Inton) 


Tota 1 Students' 


145 


129 


274 



As In thjB first year, two measurement approaches were utilized to assess 
student achievement in reading. The criterion referenced CAM tests developed by 
the 5th grade teachers of the experimental group and the norm referenced 
Metropoirtan Achievement Test (MAT) in reading were administered to both student 
groups during, the first week in June, 1973. The second year design employed only 
post tests. The fifth grade reading program employed the saipe basic CAM evalu- 
ation design as had tlie 4th and 6th grade ' programs the previous year. There were 
eight criterion referenced tests administered to the experimental group at regular ' 
Intervals throughout the year. These CAM tests 

2S7 
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had beon revi^d, based upon' Ito,. analysis data provided during tho flr=t year 
of the project. It was the result of the Item revlsieh process that gave rise to 
the socohd hypothesis under Investigation In this study. The teachers o. the 
exper.^ntal group having had a year's experience working with the criterion 
..asures, were ahle to refine the criterion test ,te.s to ,„ore accurately assess 
student achievement of the program objectives. This assu.pt-lon along with the 
inconclusive results In year one led the researcher to^SI^ I eve that the criterion 
^asures would beconie more sensatlve to the effects of a program aver ti™. 

,n the third year of the study the same basic experlmehtal-conttol design 
was used. However, the data collect,^ was expanded to Involve as many students 
as possible and to assess bacKground lnW.atlon on both populations o, students. 
The sar^ two schools remained K-5 buildings as In year t„o and both the entire 
fourth and flft. 'grade students in both schools were chosen as the sample pop- 
ulation. The fifth grade students In the experlrental group were Involved In the 

T=.hio & HRDlcts the populations Involved. 
CAM program the previous year. Table 4 depicts xne h 

TABLE 4 



.f. 



CAM Study Populations - Year 3 



1 rtr^ — ' 

\th JSrade 
5t«' Grade 


Experimental 
^Nortl)v;ood),^ 


Cent ro 1 
(CI Inton) 


Total 


154 

Ml 0 


157 
Kl 


291 
282 


f 


573 



populations of students were sfmllar, a questionnaire was sent to the parents of 
each fourth and fifth grade student enrolled in both schools. (See Appendix C 
for copy of questionnaire). Five hundred and ninety-two questionnaires were mailed 
«Hh four hundred and s I x-|-y-seven returned ^ retv^n ^percentage of 78^. 

2 as 
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, ^ Responsbs to the survey questionnaire are summarized in Tables 5 and 6% 

Table 5 • ' • - 

Studeiifc Background Data - 4th Grade 



Question 1: Father's education . 


Responses 


College 
.Degree 


Sorqe. 
Qoll ege 


U.S. 

Graduate 


Some nigh 
School 


8th Grade 
Graduate 


Less Than 
8th Grade 


No 

Response 


Experimen tal 


Fr. 


% 


Fr. 


% 


Fr. i 


% 


Fr. 


Ox 


Fr. 


% 


Fr. 


% 


Fr. 


% 


24 


17% 


19 


13% 




47% 


27 


19% 


2 


1% 


1 


1% 


2 


- 


Con trol 
N=128 


15 


12% 


34 


27% 


52 


41% , 


'20 


15% 

— 0 — 


2 


2% 


2 


2% 


3 


2% 



Key: ^ Fr. = Frequency ^ . ^ 

% = Percent ^ - 

■ ^ 



Question^2: Mother* s education - „ - . ' 


Responses 


'Attended 
College 


High School 
Graduate 


Some High 
School 


8th Grade . 
Graduate 


Less Than 
8th Grade 


No y 
Response 


Experimental 
N=141 


J Preql 


% 


Freq. 


% 


Freq. 


% 


Freq. 


% 


Freq. 


% 


Freq. 




25 


18% 


' 84 


60% 


. 23 


16% 


' 4 


3% ' 


\ ' 


1% 


4 




Control 
N^127 


25 


20% 


81 


64% 


15 


12% 


4 


3% 


1 


1% 


„ i 





Key: Freq. = Frequency 
% = Percent 



/ 1 



Question 3: Is mother working? ' ' 


Response 


Yes No „ 


Experimental 


Frequency 
40 ^ . 


% 

29% • 


Frequency 
100 


% 

71% 


Control 


41 


32% 


. 86 


68% 



Percentages are rounded to nearest percent. 
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Tabic 5 
(Continued) 



' Question 4: 


Number of siblings 






















. . Responses 


None . 


One 


Two 


Three 


Four 


Five 


Over Five 




Fr. 


% 


Fr. 


% 


Fr. 


% 


Fr. 


% 


Fr. 


% 




% 


Fr. 


% 


Experimental 
N =^ 142 


3 


2% 


26 


18% 


59 


42% 


33 


23% 


13 


9% 


6 


4% 


2 


1% 


Control 

N =' 127 


7 


6% 


34 


27% 


36 


28% 


26 ^ 


20% 


11 


9% 


10 


8% 


3 ^ 


2% 



Key: Fr. = Frequency 
% = Percent 



Question 5: Number of Qther schools attended 

r 


Responses 


No other 


One 


Two 


Three 


More than 
Three 


No 

Respo 


nse 


Experimental 
N 142 


Freq. 


% 


Freq. 


% 


Freq. 


% 


Freq. 


% 


Freq. 


% 


Freq. 


' % 


61 


43% 


62 


44% 


13 


9% 


2 


1% 


1 


1% 


3 


2% 


Control 

N = 127 


86 


68% 


32 


25% 


8 


6% 


0 




0 




1 


1% 



Key: Freq. = Frequency 



% = Percent 



Question 6: Was student's attidute toward reading' changed positively? 


Responses 


Yes 


No 


No Change 


No Response 


Experimen tal 
N = 141 


BFreq. 


Perqent 


Freq. 


Percent^ 


Freq. 


Percent 


Freq. 


Percent 


105 


^ 74% 


12 


9% 


23 


16% 


1 


1% 


Control j 

■N- =/i27 


65 


51% 


16 


13% 


44 


35% 


2 


2% 
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Table 6 ^ , V 4: 
Student Bac/cgroujid Dat^ - 5th 'Gxkii^e!: 



Question i ; 


Fa ther ' s 


Education 
















I. 






Responses 


College 
Degree 


Some 
Col l ege 


U.S. 

Graduate 


pome High 
School 


Mth Grade 
>G?:ra<?uate 


Lqss than 
8th Grad^ 


m , 

Response 




Fr. % 


fr. 


% 


. Fr. 


% 


Fr. 


% 


Fr. 


% 


Fr. 


% i 


Fr. 


% 


h^perimental 
\ iV « iOi 


16 16% 


' 20 


20% 


40 


40% 


18 


18% 


5 


5% 


1 

— ^ 


1% 


1 


1% 


'\ • ' 

Contrbl' 

N « iOi 


20 20% 


19 


19% 


39 


39% 


16 


16%' 


4 


4% 


2 


2% 


1 


' 1% 



Key:0 Fr. - Frequency 
% = Percent 



/ 



Question '2: Mother's Educatiop . 


Responses 


Attended 
College 


H. School 
Graduate 


Some^ High 
School 


8th Grade' 
Graduate 


Less than 
8th Grade • 


No 

Response 


Experimental 
iV « 99 


Freq. 


% 


Froq^ 


% 


Freq. 


% 


Freq. 


% 


Freq. 


% 


Freq. 


• Xk 


13 


13% 


62 


62% 


20 


20% 


3 


3% 


0 




1 


1% 


Control 

N = 117 


18 


17% 


60 


58% 


21 


20% 


4 


4% 


b 













Key: Freq. = Frequency 
% = Percent 



Question 3 : Is mother working? 




RespBl^t^s 


Yes 


lio 


• V ■ 


Experimental 
N ^ 98 


Frequency 


4 

Percent 


Frequency 


Percent 


f ~ — ■ 


31 


32% 


67 


68% 




Control 

N = 101 


27 


27% 


74 . 


79% 

1 
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Table 6 
(Continued) 



ibaestion 4: 


Number of siblings 






















Responses 


None 


One 


Two 


Three 


Four 


Five 


Over 
Five .J 




Fr. 


% 


Fr. 




Fr. 


% 


Fr. 


% 


Fr. 


% 


Fr. 




Fr. 




Experimen tal 
N - 105 


• 4 : 

■ 


4^ 


28 


27% 


40 


38% 


15 


14% 


11 


10% 


7 


7%; 






Control 

N = 101 


2 


2% 


19 ' 


19% 


30 - 


30% 


23 


23% 
' #- 


20 ' 


20% 


4 


' 4% 




3% 



key: Fr. = Frequence! % / 
X = Percent 



Question 5: Number of bther schobls attended 



Responses 


No^<Q^er ' 


Or^ 


Two 


Three 


More than 
Three 


No 

Response 


Experimental 
N ^ 99 


Freg. 


% 


Freg. 


X 


Freg. 


' % 


Frog. 


% 


Frag. 


% 


Freg. 


X 




47% 


42 


42% 


9 


9% 


1 


IX 


0 


0 






Control 

N = 102 


58 


.57* 


34- 


33% 


7 


7% 


1 


1% 

t 


1 


IX 


1 


IX 



Key: Freg. = Freguency 
X Petcejsit 



Question 6: 


Nas student* s attitude 


toward reading changed positively? 


Responses 


, Yes 


No 


No Change 


No Response - 




Frbq. 


Percen t 


Freq. 


Percent 


Freg. 


Percent 


Freq. 


Percent 


Experimental 
N ^ 99 


1 62 


62X 


13 


13% 


23 


23% 


1 




Control 

N = 102 


59 


58% 


15 


15% 


27 


26% 


1 


3 

1% 



Key: 'Freg. = Freguency 



ERIC 



292 

-215- 



The fami ly background of the 'students In both the experimental and control 
schools was found to be similar. There v/as a tendency tov/ard si Ightly smaller 
'families, fn terms of the numbei; of children. In the pop^latlbn served by the. 
experimental school (Question 4). The families In the experimental school also 
are somewhat more mobile than those In the .control school population as evlden.ced 
by the number of other schools attended (Question 5). Responses to,q(}8stlon 6 ' 
will be discussed In Section V. 

The same, ach levement measures wqre utilized In the third year as In the 

* > - 

previous years, both the CAM tests and the MAT. Again the C^AM. tests had undergone^ 
another revision prior to 1;he start of the school year. The same teachers as In 
previous years were responsible foj the revisions based upom Item analysis data 
received at fl^e Qn<J of the school year (June I973h In this year of the study all 
students were pre and post tested with both the appropriate level CAM and Met- 
ropol Itan Achievement Tests. Thus, student achievement galps In reading W6re de- 
termined as measured by both normative and criterion referenced t^sts. \^ 
> 

From the experience of the first two years of utilizing the CAM system, the* 

teachers In the Northwood School (Experimental Group) determined rthat one set 

> ■ ♦ , 

of objectives and criterion measures was Insufficient for all students; at any one 

* grade level. During the second year of Implementation two sets of objectives and 

criterion, measures were developed foi: both ^the fourth and fifth grade reading 

programs. In ef-fect. there were four CAM courses covering the two year span from 

beginning fourth grade to'adva'nped fifth grade. Thus a more continuous progress 

curriculum was established. This enabled the students to progress from^one set 

of objectives (course) to another at'their Indlvldaul learning rates. 



293 

O -216- 



In the third year of the study, there were a number of students Instructed 

f 

at both fourth grade levels. At the fifth grade level -only one of the tv/o levels 

\ 

of objective was used for Instruction. Thus, three sets of CAM criterion reA 
fereneed^ tests were administered to the fourth and fifth grade In both schools \ 
on a pre and post basis. 

IV. Data. Analysis . 

For each of the three years In which ach levement 'data' were collected for both 
experimental and control 'groups of students mean scores on each of the four group 
measures were calculgte^. T tests were .employed to determine statistical signifi- 
cance of differences between these mean scores. Tables displaying the results of ^ 
the- data analysis are contained In Appendix A. In addition-; charts depicting 
criterion referenced score distributions of sub populations of'^tudents are found 
In Appendl X B. . - , 

Comparisons of reading achievement betw^qn the experimental and control - 
groups of students In the first year produced ,mlxed. Inclusive results. As was 
Indicated earl ler, the 4th grade students In the experimental grOUp In year one 
had a statistically higher pre- Instruction scorp on the MAT than the cSntroI group 
while there was no statistically significant difference bety/een the 
two 6th grade groups. Thus, the 4th graders In the experimental group started out 
with a higher achievement level than the control * group. ^ 

The resuj^s of the Metropolitan Achievement, Tests (MAT) In reading which were 

0 

administered to students in the experimental and control schools at the end o.f the 
year were similar to"l^ pre-lnstructlbn results, The mean standard score for the 
fourth grade experimental group. was fbund to be significantly higher than the 
control group when a T test of significance was employed. The T statistic was 
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Calculated as 2.0819 which was significant at the P< .05 level of error. Though 
no statistical test of, significance was employed, the grade equivalency score 
for the experimental group was three and a half months higher than the control 
group,' Though the control group had a mean standard score two points higher thap 
the experimental group at the sixth grade level, 11-was not^ found to be statls- ^ 
tically significantly higher. The T statistic computed has. a value of rl.0526 
which was not significant at any acceptable level of error. The control groups 
had a slightly higher mean grade equivalent score thaa .the experimental group. 

The results of -post- 1 nstruct Ion CAM testing In year one Indicated the same 
pattern as the MAT results. The CAM tests at the fourth grade level rr^asured 
twenty-one different reading objectives with twenty-one .d I fferent test Items. 
The experimental group of fourth graders had a mean scorb of 16.41 Items correct; 
whereas, fhe control group had a mean score of 13.31 Items correct. A T test 
for statistical signlf Icartce was employed producing a value of 7.56. Indicating 
a significantly higher mean' score for the experimental group at the P< .01 level 
of -error? The sl>#h grade CAM tests consisted of seventeen Items on each form 
measuring all seventeen read I>ng course objectives. The experlrtental group had a 
mean score of 10.75. The T statistic computed resulted In a value of .223 which 
Indicated there was no statistically significant difference between the mean scores 



at -the sixth grade. , 

Further data analysis of thq' CAM test showed marked differences between the 
fourth grade experimental and control 'groups, when teacher sub-sectloni o^ each 
course. were examined. The results are presented In Table 7. 
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^- • TABLE 7 '° 
Moan Scores of Teacher Sub-^eotlons for CAM Tests - Grade 4 









Expenli^ntal. 


Control 








£jf5roup\ 


Group 


Sub-section 1 






^^•9.6 


12.15 


Sub-section 2 






14.12 


12.86 


Sub-section* 3 






16.73 


13.37 , 


'.Sub- sect I on 4 






17.10 


13.82 ' 


Sub^sectfon 5 






19.34 

* 


14.23 



Three of the class sections In the experimental group had mean scores of the 
CAM test higher than the highest class section of the control group. In one -sub- ^ 
section (mean score 19.34) ten students had a perfect score. Graphs of all • ' 
studeat scores on the CAM test can be found In Appendix B. 

Consistent wTth the overall .comparison at the sixth grade level between the 
vexperlmentaT and control group uS'Ing criterion referenced measures Is the comparison 
class sub-section. The data Indicate ybry little variation between groups, 



TABLE 8 

Mean Scores of Teacher Sub-sections for CAM Tests - Grade 6 







Experimental 


Control 






Group 


Group 


Sub-sect,Ion 


1 


10.07 


10.27 


Sub-section 


2 


10.80 


10.93 


^ub-sect Ion 


3 


10.94 


' M.07* 


Sub-section 


4 

r 


11.41 





In year two only the students who had been 4thv graders the previous year (5th 
graders In year two) were Included In the study. Also^ only post- Instruct Ion measures 
were administered to both groups (experimental group was pre-tested with the CAM 

20'6. 



tests). 
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The CAM tests consisted of thirty-four Items measuring 34 separate instruc- 
tional objectives In reading. The experimental group had a mean score of 25.2 
< Items correct] v/hereas, the control groi\p has a mean score of 23.1 Items correct. 
A T test for Stat Istica I significance was employed producing a T value of 3.0882, 
Indlcatlng^^slgnlf icantly higher mean score for the experimental group at a 
P< .01 I ever of error. 

The results of the MAT In reading Indicated a mean standard score of 83.0 
for the experimental group and a mean standard score ot 80.9 for the control 
group. The T statistic calculated H^s 1.4286 which was s I gnl f I cant at the'P< .10 
level of error but- not at the .05 level. 

Although the actual differences In achievement were statistically significant 
the magnitude of the difference is not exceptionally large, especially the re- 
suits of the standardized MAT. A score value of 2.1 difference In mean scores 
batween groups for both types of measures was. obta I ned. .' The variance of student 
scores on both tests for both groups Is also similar. The distribution of scores 
on the CAM tests are presented in Charts 5 and 6 found In Appendix B. One can 
observe that student scores for the experimental group <are skewed toward the 
maximum to a slightly greater extent than the scores of the control group students. 

This same group of students tested In a similar manner the previous year had 
* approximately the same results. The two year comparison of post-instruction mean 
scdres is contained In Table 9. • ^ 

TABLE 9 
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Two Year Comparison of Mean Scores on CAM and MAT 



. / ' 


EXPER 


MENTAL GROUP 


CONTROL GROUP 




4th 




5th 


4th 


5th 




Grade 




Grade 


Grade 


Grade 


CAM Score 


16.41 




25.2 


13.31 


23. 1 


MAT Moan Standard Score 


65.9 




83.0 


63.0 


80.9 


Sample Popu lationgiSi ze 


128 




140 


88 


• 129 
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In year throe of the study the curriculum had. undergone a major revision. 
There were now four sets of course ^object Ives and criterion measures for tKe two 
grade levels. The design was more rigorously employed in year three with pre and 
post assessments made to the experimental and control groups at both 4th and 5th ^ 
grade levels. Three of the four course curricula and sets of CAM measures were 
used in the third year. Since three different sets of objectives (3 CAM courses) , 
were employed during the year, comparisons between the experimental and'control 
groups were made for all three courses. 

AT statistic was computed to test the difference la grade equivalent mean 
scores on the MAT pre- tests for both fourth and fifth grade students. (Tables A-9 
and A- 10 Appendix A). The T values for the comparisons between experimental and 
control groups at the fourth grade level was .000 and at the fifth grade level was 
-.200, neither of which was statistically significant. Thus, there was no slgn- 
fflc^t dl\^e rence In reading achievement between experimental .and control groups 
at the beginning of year three as measured by norm refbrenced tests for either 
fourth or fifth grade students. ' 

The p«>st- Instruction comparisons of achievement measured by the MAT for fourth 
and fifth grade students are reported In Tables 10 and IK 



TABLE 10 

Comparison of Post-test Mean Scores on MAT - Grade 4 
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0Ji 

Mean Standard Sd6re 
Mean Grade Equivalent Score 
G.E.' Standard Deviation 
Sample Population Size 



Experimental 
Group 


Co'ntro 1 
Group 


73.?l 


72.04 




5.0 ^ 






. 1.6 


1.7 ] 




132 


157 
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TABLE 1 1 

I 

Comparison of Post- test Mean Scores on MAT 



- Grade 5 





*• 


Experimental 
Group 


Control 


'Miean 
Mean 
G.E. 
. Samp 


Standard Score 
Grade Equivalent Score- 
Standard Deviation 
le Populist Ion Size ^ 


81.24^ - . 
6.1 ' 
' . J. 9 

136/ ( ' 

» ■ . ' 


80.66 
6.0 
1.8. . 
141 



The calculation of the T statistic for'both fourth^d ^l^+h comparisons of 
6.E. mean s,cores produced the following respective values, 4th = 3.50 and 5th =.223^- 
At the fourth grade [ev6l the.Tvalue was statistically significant at the PC" .01 
level of error. Th~us the experimental group had a higher reading achlevetnent level 
as measured by the Metropolitan Achievement* Tesf at the end of the school year, 
although the mean .standard scora was only 1. 17 points higher. * At the fifth grade \ 
level there was not a' statistical ly significant difference In the post- test mean 
grade equivalent scores between the experimental and control group. 

The following set of tables, 12, 13, and 14 display the results of the analyses of 
CAM test data for year three. Again, both pre and post comparisons are made be- 
tween the experimental and control groups for each of the three CAM courses. 



TABLE 12 , 
Comparison of Pre- test and Post-test CAM Me 

Grade 4 - (Course 409) 



ores 
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Pre- test Mean Score 

Pro- test Standard Deviation 

Sample Population Size - 

/ 


Expev|"l mental 
Group 


Control 
Group 


10.3 
4.8 

40 - 


lk).56 
3.5 
44 


i ■ ' ■ . ■ ' . ' 

Vest- test Mean Score 
Post- test Standard Deviation 
Sample Population Size 


I9»3 ^ 
3,7 

42 


10.2 
3.9 
19 
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The CAM tests for course, 409 had 24^ test items on each form measuring twenty- 
four different course objectives. At the beginning of the year, the pre-te^t 
mean scores for botf> groups was almost Identical .with 'the experimental group, moan 
score of 10.3 and the^control group mean score of 10.56. The T statistics cal- 
^ cu.l a ted produced a value of -.280 which was not statiatlcaHy significant. However^ 
the post mecisure at. the end of the year I>ndlcated a mean score of 19,3 for the 
experimental group and 10.2 for the control group. The ca lc,ulated T value was 
8.270 which Is statistically signl f leant at the PC .01 level of error. On the aver- 
age, the students In the experimental group achieved nine more reading objectives 
during the ypar than the control group. ) 

TABLE 13 . , I 

r ' ' ' 

Comparison of Pre-tests and Post-tests CAM Mean^Scores - Grade 4 (Course 419) 



Pre-Jest Mean Scores 
' Pr^-Test Standard Deviation 
Sample Population Size 


Eikperirnentai 
Group 


Control. 
Group. 


/? 13.0 '. . 
4.9 
115 


^ 15. 16 
416 
137 


Post- test Mean Scores 


20. 1 


14.7 


PQsf-rtest Standard Deviation 


5.1 


4.8 


Sample Population Size 

»^ — • '■ • 1-! • ' 


125 


163 



The CAM tests for course 419 had 27Mtems measuring 27 different course ob- 
Jedtl vps. on each of the eight test forms. The pre-test results Indicate a mean 
score of 13.0 correct Itemp for .the experimental group and 15.16 Items for the 
control- group. The T statistic had a calculated value of -3.600 which was sig- 
nificant at the P<, .01 level of error. 

At the end of the year tfi) post- test results Indicated a mean score of 20.1 
correct Items for the experlmenta I group and 14.7 Items for the control group. 
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The"value of'the T s+atis+ic was-Q.OOO w-.hich was significant at the P< .01 level 
of error. Though the experimental group started the' schooj year ftith a statis- 
tlcally significant lower achievement level In reading than the control group, 
as measured by the CAM tests, they achieved a significantly higher level of 
achievemenf at the end of the year. There was an averag^^ain of* over seven 
objectives by the exp^rimental group w|ii le none at al l for the control group. 

; TABLE 14 

Comparison of Pre- test and Fbst-test CAM Mean Scores - Grade 5 (Course 509) 



Pr^-test Mean Score 

Pre- test Standard Deviation 

Sample Population Size 


.Experlmen'ta 1 
Group 


Control 
Group 


> 21*. 6 
5.4 
112 


20.96 
5.9 
Ml 


Post-test Mean IcOre 


25.6 


22.7 




'Post- test Standard Deviation - 




5.6 




Sample Population Size 


108 


114 


♦ 



Ther6 were thirty-seven Iterre on the CAM test forms for course 509 measuring " 
31 different. course, objectives • The p re-test mean -pcores were 21.6 for the ex- 
perimental group and 20.96 for the control group. The T statistic calculated had- ^ 
a Value of .850 Indicating no statlsticardiffejreQce In the mean scores between ■ 
the groups.^ At the end of the school year the post-test, mean scores were 25.6 
for the experimental group, and 22.7 for the controJ group. Employing a T -^est 
to determine whether the di fference between mean scores was statistically sign-. 
; IfJcant^ produced a T value of 3.920. This was" statistlca^l ly significant at the" 
P< .OJ level of error. The fifth grade students In the 'experimental group had \ • 
gained significantly more than the control group. 

The third year data was also analyzed by teacher sub-sections of the courses. 
Tables A-ll, A-12, A-13 In Appendix A display the mean scores by teacher su^-sect Ions . 
for the th^ree experimental groups* This analysis was not possible for the control group. 
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since data was not submitted by tea^sher sub-section. 

V. Conclusions and ImpllcaVlons / ^ ' 

. Three years of data on student achievement In reading v/ere coH^cted and 

analyzed to provide an emperlal base to test the hypotheses under Investigation; 

H^: There is no significant difference in the increase in student ^ 
achievement betwe.eii those students involved in a program using 
a CAM system and those students not using a CAM system. 

H^: There is no significant difference in the achievement levels of 



students as measured by either criterion referenced tests (CAM) or 
standardized norm referenced tests (f^ietropol itan Achievement Test). 



A clear definitive answer \s not -readily appareat for the first hypothesis, 
however, the data support rejection of the second hypothesis^ A specific defini- 
tion of student learning or student achievement is needed to Interpret the i'lndlngs 
of this study. If student achievement in reading Is defined as the successful 
.attainment of the instructional objectives that were developed for each reading ^ 
course, then the CAM criterion measures are the more appropriate assessment of 
student learning^ However, many educators still prefer to define and assess., student 
achievement in mope universal terms; i.e., standardized nopm referenced achievement 
tests. , • 

When eijrpfoying the definition of student achievement as the successful- attainment 
of the Instructional objectives of /tjn^^ourse, then the appropriate evidence to 
consider is the ,resuj+ of the CAM tests. The data indicate that the experimental 
groups which were provided CAM evaluation information at regular Intervals thr6ugh- 
out the course did somewhat better l/i reading achievement than the control groupis. 
If a nationally normed standardized test Is the definition of achievement then 
the results of the Metropolitan Achievement Test in reading is the appropriate 
criteria to be examined. When the reading achievement of the two groups is com- 
pared on the MAT, no clear direction is evidenced, from the data. In one case 
(year 2, fifth grade) the^experlmenta I groijp had a statistically significant 
higher mean score on tbfe MAT." 302 
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In another (year 3, fourth gr^de) the control group had a higher rnean score. In 
most cases over the three years^ however, there was no signiflca/it difference be- 
tween the experimental and control groups on the MAT In reading. Thus, It q^ould be 
concl uded that the use of Cornprehensive 'Ach ievement Man i tori ng has no major Impact, 
either positively or negatively upon student achievement as defined. by standardized 
norms*. 

This finding is not surprising since no attempt was made at the initiation of 
the project to Increase reading scores of students on standardized tests. The 
Intent was to Increase student reading achievement In those skill areas defined by 
a set of Instructional objectives developed specifically for the students Involved 
In the West Seneca Intermediate reading program. In this effort tne^. CAM system 
seems successful. 

It also can be concluded that the criterion referenced measures used in the 
CAM system are more sensItWe Indicators of student achievement than norm refer-* 
ehced tests. If .educational decision makers want eva I uation data on -the effect- . 

' Iveness of their locally developed curricula, they are more likely to detect pro- 
gram strengths and weakness employlTig criterion referenced measures than standard- 

^ ized norm referenced tests. It should be noted that by the third year of the study 
the program objectives and criterion test Items had been through twQ revisions. 
This fact along with the statistically significant differences between the experi- 
mental and control groups supports the contention that well developed criterion 
tests are more appropriate measures of the intended outcomesfof an Instructional 
program. 

In all post-test compar I sons? between the student groups using CAM and those not 

usilng CAM, with the exception of the year I, sixth grade group, the student achleve- 

h 

5 

ment of reading skill objectives was significantly higher for those^ usl n.g CAM 

The data indicate that the experimental groups which were provided CAM evaluation 

Information at* regular Intervals* throughout the course increased their 
5 

See tables In Section IV "Data Analysis^' or In Appendix- A for results obtained 
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reading skills to a greater extent th^the control groups. The graphs depicting 

» 

year end scores on CAM tests for both the experimental and control groups are 
found In Appendix B. In comparing the distribution 6i student scores It Is evident 
..that more students score at the upper ends of the scale In the CAM courses than 
in the control groups. ^ 

In examlnlng class section comparisons (Tables for year one and for year three) 
It becomes .Evident that high achievement^ course objectives Is (>osslble using a 
CAM syst^m. Though It Is possible to Improve student achievement, simply Installing 
. a. CAM evaluation system will not automatically guarantee Increased achievement. Some 
of the sub-septlon mean scores In both years, one and threel attest to that. 

Although^ the results of the CAM testing Indicate significant differences In 
achievement levels between most of the, ^xperlmenta I and conrrol groups, the MAT 
results given at the same time to the same groups more often than not did not 
Indicate a si gnl f leant dl fference. The gain In achievement of the reading skill 
objectives was detected by the CAM tests while noi" necessarily by the MAT tests. .< 
' Since the higher achievement levels of the experimental -groups was evidenced 
In all three years of the study and the fact that the fifth, graders In year two 
(fourth graders; year one) maintained a higher achievement level after two years 
In the program. It would appe^that the achievement Increase is not due to the 
"Hawthorne Effect" alone. After three years of analyzing results. It appears 
that the CAM ^valuation system does have a positive effect upon student achieve- 
ment In reading, although the magnitude of the achievement Increase Is not large. ^ 
One could conclude that the CAM system has the potential to .slgnl f leant ly Improve 
student achievement If utilized to Its fullest, 
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V/hen teachers InvplWd In the experimental ^situation (designed and usedj the , 

^ I ' 

CAM system) were asked for their Impressionistic evaluations and suggestion 

factor was emphasized. Teachers* did not have enoiigh time to examine In det<i 

the analysis reports provided t)y the CAM system. They believed that lf^thejr 

more tlm^ to Jointly analyze the CAM test results, they could have .greatly 

proved their Instructional decision making. They would have^ been abl 

adequately diagnose both Individual and group learning problems and |p 

wea^knesses. The analysis of ythe data by teacher* sub-sectlohs sup{ipH 

tentlori that when the data Is understood and utilized by classroom t 

achievement gains result. Thus> the absence of dramatic differences 

between the experimental and control groups after three yeafs may be 

adequate use of the analysts Information provided by the CAM 'system. 

had spent more time reviewing the test results cooperatively with* their ^ 

colleagues and with the stbdents, the effect upon student achievement may h 

been greater. ' 

It would also appear that the use of a CAM evaluation system has some^ 

, tlve motivational Impact upon students. The^^results of Question 6, fable^ l 

lndlcate;e(^uch larger percentage of parents of students In the experlmenta 

group'^tetl^eve their children's attitude toward reading has become more posl 

than those In the control group. A positive attitude toward reading may well 

have a long range effect upon Increased achievement of readjng skills. 

Probably the clearest Implication of the study Is the need to replicate It on 

a much larger basis. Future studies should Involve a broader sample of teachers 

and studentp and many different subject areas. Related Variables such as teacher 

training, frequency of CAM testing and time allocated to data report analysis 
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ought to be examtned. It may be that the. potential benefit of a CAM system could 
be heightened substantially with a small/Incremental Investment In In-service 
education and periodic-released time for teachers. 
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TABLES OF TEST RESULT ANALYSIS ; * 
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Tabre.A-l V ' 
Comparison of Mean Scores on CAM.Post tests - Grade 4 Cyear I) 



•1 ' J . 
/ * ' 


Experimental 
Qroup 


, — ■ • 

cfntrbr • 
_||roup 


Mean/' Sgope.. 
. . Standard Dev tat Ion 

Sample Popuf.^itlon Size 


16.41 . 
3.50 / » 
155 

— — ^ 


f 13.31 
\ 3.60 

i 143 



Table A-2 

Comparison of Mean Scores on CAM Tests"- Grad^^6 (year I) 



Mean Score 
• Standard Deviation , 
-Sample Population Size 


1 . 

Experimental /;, 
Group . 


Control 
Group 


10.84 
2.80 
128 


10.75 
2.99 
88 



Table A-3 ' " , " ^ 

Comparison of Mean Standard Schores on the .Post MAT - Grade 4 -(year I) 



■/ ' '■ ■ * ^ ^ ' 


■f ) ' ' 
Expelrt mental 

Group ^ 


' Control 
i Group 


Mean Standard Score / 
^ Standard toevl^tlop ( 
Sample Population^ Size \ 
Mean Grade EqUlva^jnt Score 

— — 


74 
11.62 
154 
4.64 


f 71.2 
; 12.85 
• ' 1 39 
4.28 


• \ . ; Table A-4 






ComiarlsOn'of Mean Standard Scores on the MAT - GradO 6 (year 1) 




Experimental 
Group 


I Control 
\Grofeip 


Mean Standard ^Sco re 
Standard Deviation . - 
Sample Population SIXe 
Mean Grade Eqvlvalmt Score 

1 t ■ ' — ~ oTTTT^ 


88 
13.15 
131 
€.8" 


90 
I4\07 
88 
7.0 
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Table A-5 

Comparison of Gain Scores on the MAT - Grade 4 (year I) 



V 

V 


'TT- ■ » 

Experimental 
Group 


Control 
Group 


Pre Mean Standard 'Wore \ 
Post Mean Standard Score ' \ , 
Gain Score 


' 65.9 
; 74.0 * 
8.1 


63.0 
71.2 i 
8.2 

/ 


\ :. " 
\ . 

Table A-6 




/a 


Comparison of Gain Scores on the MAT - Grade 6 Q^ear ff 






Experimental 
Group 


Control 
Group 


Pre* Mean Standard Score 

Post f^e^ Standard Score * ' , 

Gain Score 


83,4 
88.0 
4.6 


84. i 
90.0 
. 5.7 


Table A-7 ;^ 






xi Comparison pf Mean Scores on CAM Tests f 


5th grade (year 2) 




Experlmenta 1 
Group 


Control 
Group 


Mean Scores ' ' 

Standard Deviation 

Sample Population Size 0 

■ * 


25.2 
5.1 
140 


23.1 


3 Table-A-8 '■. 






Comparison of Mean Scores on MAT r 5th 


Grade (year 2) 






Experimental 
Group 


Control 
Group 


Mean Stanj^Jard Scdre 

Standard Deviation 

Sample Population Size ^ 


83.0 • 
12.1 
145 


80.9 
12.0 
128 


i 
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' ... ' T^blo A-9 . ■ ' 

Comparison of Pre- test Mean Scores on MAT - Grade 4 (year 3) 



♦'■ 
.» *' 




Experimental 
broup 


Control 
Group 




\ . 

Mean Standard Score 

<Olean Grade Equivalent Score 

G.E* Standard Deviation ^ 

^rirhnlfi Pnnii 1 ri+ Inn STzfi 


63.7 
3.7 
1.34 

134 


64.0 
^.7 
1^.6 
139 




\ ' ' Tabfe A- 


10 


•> 


• 


V 

Comparison of Pre-test Mean Score 


on MAT - Grade 5 (year 3) 

» 






Expe^mental 
Group J 

^ - . 


Control 
Group 




Meiii\ Standard Score 
Mean Grade Equivalent Score 
G.E. Standard Devi at I ort 
Sample Population Size 


- 75.2 / , 
5.32 \ 

1.5 ^ 

141. 


75.4 
5.36 
1.53 

138 . 



Table A-ll 

Mean Scores of Teacher Sub-sections for CAM tests - course 409 (year "3) 







Number of 




Mean Score 


Students 


Sub-section 1 


I2;3 


4 


Sub-section 2 


19.8 


1 1 


Sub-sectl-on 3 


20.1 


17 


Sub-section 4 


20,3 . 


10 


Total Possible Score 


. 24 
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Tabic A- 1 2 



(year 3) 



Sub-section 1 
Svb-sQction 2 
Sub-section 3 
Sub-section 4 
Sub- section 5 
Sub- section 6 
Sub-section 7 
Sub-section 8 
Sub-section 9 
Subsection 10 
Sub-section 11 
Total Possible Score 



Mean Score " j 


Wujnbor of SLuaani. .. j 


8.8 1 


5- ' 


^12.6 1 


8 


16.3 




17.1 


10 


18. A 


10 


19.2 


j 23 * • 


21.4 




23.3 


j 16 


23.9 


1 


25,2 




25.8 


1 " 4 • 


27 







Table A- 1 3 

■ ' ' ^ f«r. rAM rests - Course 509 Cyear 

. wean Scores of Teacher Suh-sectiorjs for CAM 



Su2>-section 1 
Sub-section 2 
Sub-section 3 
Sub-section 4 
^Sub-section 5 
Subsec^on 6 
Sub-section 7 \^ 
Sub-section 8 



Total Possible Score 
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APPENDIX B 



Graphs of Student 



CAM Test Scores 



I* 



} ■ 
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STUDENT BACKGROUND DATA QUESTIONNAIRE 
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SCHOOL 
_ STREET 
,NEW YORK 



, 1974 

Dear Parent : . " 

Please provide information requested on the following- form. This information is 
needed for your, child's folder which follows hira throughout his schooling. 



This information is only for school records and is confidential. 



Principal 



PUPIL'S LAST NAME FIRST NAME MIDDLE 



BIRTHPLACE DATE OF BIRTH SEX 

FATHER'S NAME BIRTHPLACE 



FATHER'S EDUCATION (a) ATTENDED COLLEGE 



^.^^ Degree (s) earned 
No 



(b) GRADUATED FROM HIGH SCHOOL BUT DID 
NOT ATTEND COLLEGE 



(c) ATTENDED BUT DID NOT GRADUATE FROM 
HIGH SCHOOL 



(d) FINISHED 8TH GRADE BUT DID NOT 
ATTEND HIGH SCHOOL 



(e) DTD n6t FINISH 8TH GRADE 
fiOTHER'S NAME • BIRTHPLACE 



MOTHER'S EDUCATION (a) ATTENDED COLLEGE 



(b) GRADUATED FROM HIGH SCHOOL BUT DID 
NOT ATTEND COLLEGE . 



(c) ATTENDED BUT DID NOT GRADUATE FROM 
HIGH SCHOOL 



(d) FINISHED 8TH GRADE BUT DID NOT 
ATTEND HIGH SCHOOL 



■ (e) DID NOT FINISH 8TH GRADE 
PUPIL'S PRESENT ADDRESS 



PHONE 
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«uw L.j.vJlf(U WITH 



FATHER'S OCCUPATION, WHERE EMPLOYED 
MOTHER'S OCCUPATION, IJHERE EMPLOYED 



NA>ffi OF BROTHERS ] YEAR BORN 

__; YEAR BORN 

; YEAR BORN 

NAME OF SISTERS [ " YEAR BORN 



YiEAR BORN. 
YEAR BORN 



?hI PMSm'^N^ """^ '"^^^^ '''' ™^ ""^^^ $CH00k0THER THAN 

— - YES 



■ NO 
IF YES, HOW MANY OUTSIDE OF WEST SENECA ' ^ 

IN YOUR JUDGEMENT, HAS YOUR CHILD'S ATTITUDE ABOUT READING - 

INSTRUCTION CHANGED POSITIVELY IN THE LAST TWO YEARS? NO 

r NO CHANGE 

DOES YOUR CHILD MA^ USE OF ,THE LIBRARY SERVICES MADE AVAILABLE?' 

, School library 

^heck those which apply Bookmobile 

Public library 

IN YOUR JUDGEMENT, HOW WOULD YOU RATE THE CHILD'S GENERAL HEALTH? 

' Poor ' 

Fair 

» ♦ ^ Good 

Excellent 

.IT IS VERY IMPORTANT TH AT YOU RETURN THIS TO YOUR CHILD'S HOMFRnnM 
TEACHER AS SOON AS POSSIBLE. 



ERIC , 
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11.0 Interview Schedules and Response Records 
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SURVEY OF COMPENSATORY READING PRTCRAMS 
_ BUILDING PRINCIPAL INTERVIEW 



Directions: This interview is in two parts. 
The first part is intended to elicit infor- 
mation about yoxir school and the students in 
it.. The second part of the! questionnaire has 
to do with* compensatory reading programs. By 
compensatory reading instruction is meant any^ . 
reading instruction provide4 to students because 
they are reading below their grade level. 



' -'I 



• 1 .) 



PART I 



PLlASE PROVIDE THE FOJ^OWING INFORMATION ABOUT YOUR SCHOOL. Answer all questions 
I with reference to the current school year unless otherwise indicated.' 



1. 
2. 

3. 



5. 



6. 



School enrollment this year - all S. 
grades, (number of pupils). 

Number of classrooms.- all grad'es. 
(Do not include offices, I 
auditorium, or gymnasium.) 



If you have a combination of 
graded and ungraded classes, 
Indicate the instructional 
organization for each grade, or 
if ungraded, the equivalent grades 
in your school. 

Number of classes and. students at 
each grade level for levels 4-5-6. 

Percent of total student body that 
moved from school attendance area 
last year. 



7a. Do ypufeel this is an accurate 

estimate? Yes, No, or approximate? 



Estimated percentage of pupils whose 
families receive public assistance. 



8a. Do you feel this is an accurate 

estimate? Yes, No, or approximate? 

9. Estimated percentage of pupils whose 

head of household attained the following 
levels of education. 



9a. 
10. 



Do you feel these ar 
Yes, No, or approxi"* 



-ccurate estimates? 
e? 



Percent of total student body that 

moved into school attendance area 12. 

last year. 

Estimate the percentage (this year) 
of pupils from families of migrant 
worfcers. / - , 13. 



J J ^ mm~^m^ ~. 

Estimate the percentage of school 
families fn each of the following 
occupational categories. 

10e. Do you feel these are accurate estimates? 
Yes, No,^ or approximate? 

11. Estimate tlie percentage of students 
of*th6 folldwing racial. or national 
origin?. I* 



'If children are bussed for racial balancej^l 
about what percentage of the total stude 
body is bussed in? 



1 



,.If children are bussed for racial balance | 
about what percentage of the total student 
body is bussed •from your school's / 

neiahbor hood iS? 
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14. Using your best professional / 
judgment, rate each of the 

^ following characteristics 
. for your school. 

15. Estimate the percentage of 
students in your school at 
each of the following grade 
levels who* are reading one 
or more years below grade . 
level according to current • 
test data./ The estimate 
should be based upon the [ 
concept of national norms 

for ^the grade for which 

you^re reporting. ' 

16. ^re there students in your t 
school who in your judgment - . , 
are in need of reimedial • 
reading instruction but * / 

; - who are not receiving such 

instruction? Estimate the 

number or proportion at 
. each grade level* 



PART il ' 

READ: , V ^ 

The next part of the interview is intended to elicit information about 
tfie compensator^^ reading program(s) in your school. By compensatory 
reading instruction is meant any reading instruction provided to students . 
because they are reading below their grade level. 

If you have more than one compensatory reading program in operation in 
your school during this academic year, s^^ace is provided is some instances 
for you to answer questions about each program individually* Some guide- 
lines for determining what constitutes "a program" for purposes of this 
survey are pK^ented below. 

1. If instructional groups (for example, grades) are exposed to materials, 
personnel, and services, in the regular classrooms that are supplementary 
to the 2;eguXar reading program, this should be considered compensatory; 

2; If a separate classroom or space is set aside for reading instruction, 
staffed by special personnel and supplied with special equipment or 
materials, such an entity shpuld<Be considered a program. 

3.. If teachers receive special training for compensatory reading 
instruction during summers or relaxed time, and that training is 
funded by supplementary sources, such training, in and of itself, 
• should be considered a program. > . ^ ^ 



17. Describe any sj[>ecial 
facilities (a^ iStaff) in 
general use among 
compensatory reading 
personnel, the classroom i 
teacher, and, others involved 
in thq reading program (e*g., 
media centers etc,)« 

18. What was/^your total (salaries 
plus materials) school per 
pupil expenditure last year? 

19. What are the total ^^unds 
allocated for compensatory 
reading in your school? 

(Note: this can be 
completed easily by 
adding up values provided 
in all programs in question 
20 that follows J 

20. The following are a series of 
questions concerning the 
organisation and intent of 
each, of your compensatory 
reading programs. Let's take 
each program in turn and 
con4>lete the questions. 

(TURN TO THE RESPONSE 
SHEET AND ONE SHEET FOR * 
EACH PROGRAM) 



332 



BP 



BUILDING FPINOIPAL IMTERVIM: RESPONSE, RECORD 



Name of iaterviewer 
Date ' J 



Length of Interview 



WW 



Interviewee Name 



District 
' Building 



Grade or CJrade Levels iicluded K 1 2 3 4 5 6 7 8 — 



Additional or Adjunct Data »to be Collected: 




Conations, dates, or other arrangements for collection: ^ 



\ Jemarksc • 



Total 

1. Enrol Imfln-f-. 



Total No.._ 
2. Ciassroc^s 



3. CSraded 
^ * Check 



Uigraded 
Check 

TJ 



gr. 6 ^ 




Graded & i 

Uagraded 

"Check 

□ 

□ 
□ 



4. 









No. Students 


gr- 


4 


^JJo.—Gliissea — 












gr. 


5 










' ■ . ' ■«. 


gr. 


6 , 




■ 



5. A moved ouii. 

6. % mpved in 
* 7. % migrant 

7a. accxarate est.? Yes 

Approximate 
8. % public assistance 

8^. accurate est;? Yes 



5, 



□ 
□ 



6. 
7. 



No. 



No. 



Approximate, 
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9* Educational attainment 



" Attended aollege 



Graduated from high school 
but did not attend college 



Attended but did not graduate 
f rpm high school 



Finished &th grade but did 
hot attend high school 



- 9- 



Did not finish 8th grade 

9a' Do you feel these are accurate 
eatiraates? Yes ^ N6. 





T. 



^. Approxijnate_ 
lOOccupational level 

Unemployed/or welfare 



Unskilled worker 

(gas station attendant) 



Skilled worker-blue collar 

(fifeman^ electrician) 
' *■ • 

Skilled worker-white^ collar 
(office, sales) 



Businesi^: management 
or ^)vm9r level 

Profession^E (doctor, 
lawyer^ .teacher) 



10. 



lOa. Do you feel these are accurate 
estimates? Yes No 



Approxiinate_ 
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NOTE 




11. Ethnic conqposltion 

Caucasian or White 

Negro or Black 
Spanish sumiLed 
Oriental 
American Indian 

Other fepeclf y 

12. % Bussed in 

13. % Bussed out 



1) 



I 



I 



I 



I 



14, School Characterlgtlcs 



Highly 

Adequate Adequate 



Inadequate 



14, 

Inadequate 



Size of physical plant for pupils 


□ 


□ • 


□ 


□ 


Condition of physical plant 


□ 


□ 


□ ■ 




Suitability of physical plant for 
program operation 

Is the n\amber of dLnstructlonal personnel 
(teachers, specialists) 


□ 


□ 
□ 


□ 

.a 


□ 


Is the nxmiber of other prpf essional 
personnel (psychologists, social workers) 




□ 


p 


□ 


Suitability (quality) of books, 
^periodicals, and other printed materials 
^3^T instruction 


□ 


□ 


□ 


□ 


Suitability (quaiytts^ 

visual mafc^rddis f or instruction 


□ 


□ 


□ 


□ 


S\iitabinty (quality) of in^inict'ion&l 
equipment for instruction ^ 


□ 


□ 


□ 


□ 
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15 

. grade 4 
grade 5 
grade 6 

l6. grade 4 
gradj^ 5 
grade 6 



Reading 
Below Qrade Level 



3 



3 



3 



3 



3 



17. Special facilities. 



15. 



16, 



18, school esqpenditure (salaries 

plu^ materials) 



• 19 • Total compensatory expenditur;e 
(salaries pl\is materials) 
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The 'State Education Department 

Bureau of School and Cultural Repearch 
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2. 



3. 



4. 



5. 



6. 



Numbers o£ teachers and students serviced 
by this program? 

Can you provide us with a time or appoint- 
ment schedule showing the names of students 
in classrooms serviced by you in your pro-* 
' gram at levels 4, 5, and 6? On this 
schedule^ would you circle any student who 
hlas not been \7\th you for the year but who 
has been seen by you over a period of 
beftween 10 and 20 weeks. If a student will 
be seen by your for less than id weeks this 
school year, draw a line through his name. 
This list will communicate as follows: 



Studei!^t name (as is^: 



Student has been 
(or will be) with 
you for more than 
% the school year 
(more than 20 wks) 



Student name (circled) 



Student has been 
, ' (or wijl be) with 

you regularly for 
no more than 20 
weeks arid not less 
than 10 weeks ' 

Student name (line drawn through) 

Student has been 
(or will be) with 
you regularly for 
less than 10 wks) 

What' percentage or number of the pupils 
in your program (4-5-6) are members 
of the following racial or national 
origin giroup? 

(Point to the response sheet) 

What is the average or typical absentee 
rate in your sessions with students in 
grades 4, 5, and. 6? 

(Daily number or percent) 

What is the % or number of new pupils who 
will be taken into your program during the 
school year in grades 4, 5, and 6? 

What % or number of pupils in grade 4, 5, 
aipti*6 will move out of your program during 
this year? 



10. 



10a. 



12. 

13. 
14. 

15. 



How far do you expect the typical 
pupil in your reading class will 
actually go in school under the 
existing program? 

(Point to Jthtf response sheet) 



If the students in your reading 
class were given whatever special o 
support or programs they needed 
throughout their years in school, 
what is the highest level that the' 
typical pu^>iJ^-would achieve? 

(Special reading programs, 
cultural enrichment, individ- 
ualized programs, etc.) 

During the school year, how many 
professldhals other than yourself 
have held your particular assignment 
with your reading classes for at 
l^east two consecutive weeks ? 

Do you orgatiize your instruction 
into groups or is it individual 
instruction? 

Describe the criteria or diagnostic 
materials used for determining group 
placement or individual program. - 

How frequently does the compositiMph 
of your reading groups change 
because of different rates^of learn*- 
ing or different skill deficiencies 
among students? 

Describe any unusual or outstanding 
features of your program at levels 
4-5-6. 

Describe the facilities used in your 
instruction. 

Please indicate what materials you 
use in your reading instruction and 
to what extent you use them. 

Indicate any additional materials 
not on the printed list. 
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16, Please list any equipment or ' 
hardware used in your reading 
program. 

(Indicate as major or 
supplementary and the 
number of hours of 
' " use per week) 

17. Please list any instructional 
materials that yo^ create for 
your, reading program, such as 
charts, ^ ^ 

(Point to the response sheet) 
and whether they are a major or 
supplementary resource. 

17a. What is' the per-monthly or 
ailnual cost of the 'self- 
created reading materials? 

1-8. For the materials tfiat you 
listed as in regula'r use in 
your reading program, what 
is the length of your use 
or experience? 

19. Briefly describe your formal 
training in the teaching of 
reading in terms of types of 
courses and recency. 
(Point to the response 
sheet) 

20* What types of inservice 

training in the teaching of 
reading have y^u had in the 
last twt) years? 

21. What is your average amount 
of preparation time for 
reading instruction pef 
week? 

22, What ts the average amount 
^ of time spent per week in 

coordinating activities 
, in your reading program / 

with other professional 

personnel, such as the 
^ classroom teacher? 
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SPECIALIST INTERVIEW: RESPONSE RECORD 



Name of Interviewer 



Date 



J 



Length of Interview 



Interviewee's Name 
Names of other staff 



District.^ 
Building ^ 



Type of teacher^ 



Grade or Grade Levels Serviced K12'34 5 678 
Title of this Compensatory Program - 



Addletional or Adjunct Data to be Collected: 



Conditions^ dates, or other arrangements for collection: 



Remarks : 



!lOXE: Complete one form for each Compensatory Reading Program 

-264- 3^j^ 



^1. Grade 4 Grade 5 Grade 6 !• ' 

Teacher Student Teacher Student Teacher Student 



2. Schedule attached? ^ 2. 



3. (a) Caucasian or White I ' 7»\ \ | 3. ' 

(b) Negro or Black 

(c) Spanish sumamed 

(d) Oriental 

(e) American Indian 

(f) Other_ 



□ 

□ □ , 

□ □ ^ 

□ □ 

□ □ 

□ □ 



3a. Do you rgel*' these are accurate estimates? 3a. 
yes ^no , a pproximate 



4. Rate absentee , | % | | I 4. 

nn 

5. Move into program during, year | % | | | - 5, 

- • I 

6. Move out of program during yearj 



LjJLJ 6. 




-3^ 



7^. Educational attainment, 'exislting qdnditioris 

Eighth grade or lower | ^ . Q 

Ninth, tenth, or eleventh grade D 

High school graduate Q 

Junior college, business school, or any * 
other post-secondary tr^lni^ng but not a * 
four year college. ''j,)- C3' 

Four year college or beypn^ 

Other (Specify) 



8. Educational attainment, idjeal^onditions; 

□ 

grade . Q 

□ 

□ 
□ 

_□ 



Eighth grade or lower 
Ninth, tenth, or eleventh 
High school graduate 



Junior college, busines*s ichoor or 
any o ther ^ pos tr secondaicy training 
but not a four year college 

Pour year college or beyobd 

Other (Specify) 

9. . Teacher absence 



10. Group or individual instruction 



10a. Criteria or diagnostic materials used 
Tests 



Other 



343 
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11. Erequency of change in grouping 



12. Outstanditfg featutes^ 



Facilities used 



344 
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14. Series Titles^ 
(Specify) 



Major 
Resource 



Supplemental 
Resource 



I \ 



Addm^Wesley 



Allyn'fit Bacon 



American Book Co» 



Benef ic 



FolLstt 



Ginn & CoV 



Harcourt Brace xx 
World 



Harper Rov;. 



Holt-Rinehart & ; 
V/inston 



Hpughton-Mif f 1 in 



ITA 
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□ 


n 








1 — \ 

□ 


B 


LJ 


U 


□ 


r — I 


\ □ 


• □ 


\ 1 — 1 

yU 


□ 


□ 


□ 


□ 




O 




□ 


□ 


□ 


□ 


□ 


r ■■■» 

□ 




□ 


□ 


□ 








□ 


□ 
□ 


□□□ 


□ 




□ 


□ 


□ 


□ 


□ 


□ 


□ 


□ 


□ 


□ 


□ 


□ 


□ 


□ 


1 — 1 


□ 


□ 


□ 


□ 


□ ■ 


□ 


□ 


U 


□ 


D 


□ 


□ ■ 


□ 


□ 
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1... . - 

' Laidlaw 



Series Titles 
(Specify) 



Lippincott 



Macraillan 



McGraw-Hill 



Merrill 



Modern Curriculum. 
Press 



Open Court 



Randdm House 



Scholastic 



Scott Poremcin 



SRA 



Major 
Resource 


Sunplementa; 
Respux^^e* c 


- □ . 


■'Vo - - i! 


. □ 




P 


□ : 


□ 


. □ J 


□ 




□ 




□ . 




□ 




□ 


' 0 li 




: n 


□ 


.-D ■•■^ 


□ 


□ ■ -. ■ -^i 


□ 


□ H 


□ 


□ 


□ 


□ i 


_ □ 


■•1 


□ 


□ ! 




□ 1 


□ 


□ 


□ 




□ 


□ • 


□ 


□ 


□ 


□ 


□ 


□ 




□ 


□ 


□ 


□ 




□ : 


□ J 


□ 


□ i 


- H 




□ 


□ 


- D 




n 


□ 


- P 




- □ 


□ 


□ 




- □ 


□ 
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l^Bt all Other software including workbooks, 
magazines, tradebooks, classroom libraries, 
films trips, movies 



Used as major^ 
resources in 
teaching reading 



Used a^ 
supplement 



0 
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346 



□ 




□ 


□ 




□ 


0 




□ 






□ 






□ 






□ 


□ 




□ 


□ 




□ 


o 




□ 


□ 




□ 


p 




□ 


' □ 




□ 


□ 




□ 


□ • 




□ 






□ 


□ 




□ 


□ 




□ 


b 




□ 


d 


- 


□ 


□ 




□ 


□ 




□ 




* 


□ 


□ 




□ 


□ 




□ 


□ 




□ 


□ 




□ 



16. Hardware 



Used -as a 
major resource 
In teaching 

□ 
□ 

-□ 

□ 
■ □ 
□ 
□ 
□ 



Used as a 
supplement 
or option 

□ 
□ 

□ - 

□ 
□ 

□ 
□ 



Amount of 
use/week 



16 



9 



17. Materials Created 



-1 



Worksheets 

Transparencies for over- 
head projector 

Slides 

Charts ; 
Tapes 

Other (Specify) 



Used as a 
major resource 
in teaching 
reading 

□ 

□ 

□ ■ 

■ □ ' 
□ 

_ □ 



Used as a 
supplement 



□ 

□ 
□ 
□ 
□ 
□ 



17a. Per-monthly or annual school cost for self- 
created materials 



monthly 



annual 



18. Length of experience with materials 
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19. Formal training; college courses 



Number of courses ^ 

Course Content 

Kew instructional 

techniques in reading 

Diagnosis of reading 
problems 

Open classroom methods 

Individualized instruc- 
tion 

Use of equipment & 
materials 

Techniques for oultural 
enrichment 

Other ; 



Wit;hin 
5 years 



□ 

□ 
□ 

□ 

□ 

□ 
□ 



20. Inservlce traijjAilg within the 
last 2 ikam^r 

I ' Within 



2 years 



Number of clock hours 
per month 

Course Content 

New instructional 

techniques in reading 

Diagnosis of reading 
problems 

Open classroom methods 

Individualized instruc- 
tion 

Use pf equipment & 
)ptiaterlals 

Techniques for cultural 
enrichment 

Other 



□ 

□ 
□ 

□ 

□ 

□ 

□ 



Before 
5 years 



□ 

□ 
□ 



□ 
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Preparation time (minutes per week) 



Coordination with other personnel 
(minutes per week) 
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CLASS 
CHARACTEltlSTICS 

How many pupils are in your daily 
eading class? 
(Haye teacher check the class 
list) 



What percentage or number of the 
pupils in your daily reading class 
have received compensatory readtHg^ 
instruction prior to this year? 
(Programs designed to compensate 
through time and effort beyond 
the daily reading program, 
usually are disadvantaged kids) 

What percentage or number of the 
pupils in your reading class 
attended some form of- prej^chool? 
(Head Start, day care, nursery 
school) DO NOT INCLUDE PUBLIC 
SCHOOL KINDERGA.RTEN. 



11. What is the percentage or number of 
pupils in your class who will move out 
of your class before the end of the 
year? . 

12. Ho'w far do you expect the typical 
pupf! in your reading class will 
actually go in school under the 
existing program? 

(Point to the response sheet) 

13. If the students in your reading class 
were given whatever special support oi 
programs they needecf throughout their 
years in ^school, what is the highest 
level that the typical pupil would 
achieve ? 

(Special reading programs, cultucal, 
enrichment, individualized programs 
etc*) 



What percentage or number of the 
pupils in your class are members ' 
of the following racial or 
national origin group? . 

(Point to the response sheet) 

What percentage or number of the 
students in your class come. from 
homes where the dominant language 
is not English? 

If the dominant language is not 
English, list those spoken. 

Indicate' the percentages or numbers 
of pupils in your class whose family 
incomes are derived from each- of the 
following occupational categories. 
(Point to tKe response sheet) 

What is the. average' or typical 
absentee rate in your class? 
(E^^ily number or percent) 

What do you feel is the major cause(8) 
of absenteeism among your pupils? 
(Ex*s: illness or pupil; illness 
in family, truancy, pupil performs 
duties at home; expulsion) 

What is the percentage or number of 
pupils in your class who will move into 
your class before the end of the year? 



PROGRAM 
CHARACTERIS^^^fi/ 




14. If your reading class ife organized 
into groups, describe the criteria 
used for detelrmining placement of 
students into groups. (e.g», 
standardized test scores, skill 
deficiencies, etc. ) 

(Write down the names of the 
standardized and other tests used.)! 

15. How frequently does the composition 
of your reading groups change becausei 
of different rates of learning or 
different skill deficiencies among 
atudents? ^ i 

16. Describe any unusual or outstanding 
features of your reading program. 

17. Please indicate what materials you 
use in your reading instruction 
and to what extent you use them. \ 

1-8. Indicate any additional materials 
not jon the printed list. 

19. Please list any equipment or hardware 
used in your reading program, 

(indicate as major or supplementary 
and the number of hours of use per 
week) 



20. Please list any instructional 

materials that yoU-create for your 
reading program, such as charts. 

(Point to the response sheet) 
and whether they are a ma jot or 
supplementary resoutce. 

20. a. What is the per -monthly or 

annual cost of the self-created 
reading ^terials? 

""21. For the materials that you listed 
as in regular use in your reading 
program^ what is the length of 
your use or experience? 



27. On average how much time'does a' . 

typical pupil in your class get for; 
^ree, non-instructional reading 
activities? / " 



TEACHER 
CHARACTERISTICS 

22. Briefly describe your formal 
training in the teaching of 
reading in terms of types of, 
courses and recency. ^ 

(Point to the response sheet) 

23. What types of inservice training 
in the teaching of reading have 
ypju had in the last two years? 

24. What is your average amount of 
preparation time for reading 
instruction per week? 

25. What is the average amount of time 
spent per week in coorddLnating 
activities in your reading program 
with other professional personnel 
(e.g., the reading iBpeclalist, team 
members, psychologist, etc.) 

26. During the school year, has "anyone 
other than yourself have held your 
particular teaching assignment with 
your reading class for at least two 
consecutive weeks? 

COUNT ^ SUBSTITUTE TEACHERS AND 
REPLACEMENT TEACHERS: DO NOT 
COUNT STUDENT TEACHERS OR 
_ . CLASSROOM AIDES 
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TEACHER H^TERVIBf; RESPONSE RECORD 



Name of Jiiierviewer 



Date 



Time Sc)ieduled_ 
Time Started 




Intervievf^ Name_ 
District 



Buildiijg^ 



Grade or Grade Levels Included K12 34567'6 



Additional or Adjunct Data to be Collected: 



Conditions, dates, or other arrangements for collection: 
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Number pupils in reading class 



Received Compensatory education 



Rceschool attendance 



Caucasian 
Oriental 



1. 



American 
Indian 



% 



Dominant home language 
, not English 



Language: 



Head of household — occupation 
Umemployed/or welfare 



Unskilled worker 

(gas station attendant) 



% 



% 



Skilled worker - blue 
caLloa^. (jPireman, 
electrician) 



Skilled worker — ^whita ! 
collar (office, sales) 



Business; 
level 



manaficnftnt 



Professional (doctor, ( 
lawyer, teacher, etc. ) 

Don't know 



% 



1 



% 



354 



8. Absentee rate (per day) 

9. Cause of absence: 



9. 



10. Move into classroom since Sept. 



% 



11. liove out of classroom since Sept. 



10. 



11. 



12. Educational attainment, existing conditions 
Eighth grade or lower ( I 
Ninth, tenth, or eleventh grade [ I 

Hiish' school graduate - f | 

Junior college, business school, 
or any other post-secondary* - 
tra^ng but not a f ouor yfear f I 
cdllege V 

Four year, college or beyond 

Other (Specify) ^ □ 

13. Educational attainment, ideal conditions 
Eighth grade or lower f I 
Ninth, tenth, or eleventh grade I I 

High school graduate \ ^ \ 

Junior college, business school, 
or any other post-secondary 
training but not a four year — — . 
college . I I 

Four year college or beyond 

other (Specify) ^ □ 



r> r- |— 

o [) a 



PROGRAM CHARACTERISTICS 
14. Grouping Criteria 14. 



Teste 



Ofcher_ 



15* Group composition change: frequency 15 , 



16. Outstanding feat\ja:*e3 of reading program l6. 



if 



35G 
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Sui>plcr.;onl'al 
Resource 



Series iW-trles 
(Specif J/) 



□ 

n 

4 



□ 

D 
□ 
□ 
□ 

n 
□ 

□ 
□ 

□ 
□ 

□ 

□ 
□ 
□ 

□ 
□ 
□ 

D 
□ 
□ 

□ 

□ 
□ 
□ 
□ 
□ 
D 

n 

D 
D 



Laid law 



•LippinciiLL 



Mecmillan 



McGraw-Hill 



Merrill 



t-^ioflerji Curriculum 
Press 



Open Court 



Random House 



Scholastic 



\ 



Scott Foresmi^n 



SRA 
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Major 
Resource 

. □ 
. □ 

- □ 

- □ 

_ □ 
_ □ 

^ Q _ 
_ □• ■ 
_ □ 

- □ 

_ □ ■ 
_ □ 
_ □ 
_ □ 

- □ 
_ □ 
_ □ * 

_ □ 
_ □ 
_ □ 

_ □ 
_ □ 

_ □ 
_ □ 
_ □ 

- a 

- D 
_ O 
_ □ 
_ □ 



Supplement^J 
Resource . hJ 

□ 
□ 

□ 
□ 

□ 

□ 

■ 

o 
o 

■) ■ B 
□ 

□ 
□ 
□ 
■ D 
□ 
□ 
□ 
□ 

D 
□ 
□ 

□ 
□ 
□• 
D 

O 

p. 
□ 



18. All other software including 
workbooks, magazines, trade- 
books j,, classroom libraries, 
filmstrips, movies. 



19. Hardware 



Maj. Sup. 
Res. Res'. 



Amount of 
uso/we(?k 



20. Materials created :-v; 
Vtorksheets 
Transpar eicies 
Slides 
Charts 




Maj. Sup, 

Res, Res. 

□ □ 

□ □ 

□ □ ■ 

□ □ 

□ □ 

□ n 

□ □ ■ 

□ □ 

□ □ 

□ □ 

□ □ 

□ □ 

□ □ 

□ □ 

□ □ 

□ □ 





□ 


□ 


■f 


u 


□ 




□ 


Q 




. : a 


D 




a 


□ ■ 




Q 


□ 




Q 


□ 




□ 






Q 


□ 




□ 


n 




D 


ni 




Q 






a 


n \ 




a 


n 




" a 


n 




n 


n 






n • 


» ■ 


□ 


□ . 




□ 





Major 

□ 

□ 
□ 



cost of self created materials 



21. Experience with major resource materials' 



ERIC 



1'^ 



Supplement 

□ 
□ 
□ 
□ 
□ 

Per Month ^ 

Per Year 
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NOTES 



22. 



•4 



•ERIC 



22. Formal training: college courses 

Within 
5 years 



Number of courses 
Course- Content' 

New instructional 

techniques in reading 

Diagnosis of reading 
problems 

Open clas&room methods 



Individualized instruction 



Use of equipment & 
materials 

Techniques for cultural 
enrichment 



Other 



□ 
□ 

□ 
□ 
a 

D 
□ 



23. \lnservice training within the 



^ast 2 years. 



Number of clock hours 
pet month 

Cout^se Content 

New instructional 

technrques in reading .J 

Diagnosis of reading 
problems 

Open classrodm methods 

Individualized instruction 

Use of equipment & 
materials 

^Techniques for cultural 
enrichment 



Within 
2 ye&^s 



353 Other 



□ 
□ 

□ 

□ 
□ 

□ 
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24, 



NOTES 



24. Preparation for reading: min. /wook;. 



25. 



25. Qpordination of reading instructi</>n with 
ot^her prof e«ssionals ; inin. /weeU '// 



26. 
27. 



26. 
27. 



'Teacher absence 



Noninstructional reading activities: 
rain./v;eek- , 



ERIC 
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Teacher hanle 



ALLUGATiON OF STUDENTS TO PROGRAM 

' Program Niimber 



(Minutes per week) 



Student 
Number , 


} Student Name 

1 ' \ 


= 




















1 




: — — ^ 












— ..^ 

t 




















- ? 1 












































• 
















































































































































































































If ^ ' 
















— ! ' '■ ^ 
















1 

i , 
















1 
















































1 ' 

1 . — 1 
















































I 
















( 
















-1 ^ - 
















» ■ 
















~i : ^ 






















































































_v 






— \ — 




















J 
















r f 


1- 














^ — 


-H — . — 


















— t\ — 

i \ 
















^^j, — 


I ' \ 
















h— ^ 






























\ L_ 




- 


















I 


















1 / 








r-| 






4- 
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— 






























— 


























^ — \ 
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CHARACFERIZATION OF PROGRAM 



Dtractor 



Program Name 



Location 



1.1 



Whole Class 



Teacher 

□□□□ 

SpQclallst 

□□□□ 

Paid Aide 

□□□□ 

Vol, Aide 

Other 
□□□□ 



Tn<;f 

Alio U t 


oup* 


/♦ 


V 




7 


7. 


\; 7. 


7. 


7. 


7. 


7. 



U2 Small Group 

Inst, Sup> 



Teacher 

□□□□ 

Specialist 

□□□□ 

Paid Aide 

□□□□ 

Vol. Aide 

□ □□□ 
Other 



% 



1.3 Individual 

Inst. Supl 



Teacher 

□ □□□ 
Specialist 

□□□□ 

Paid Aide 

□□□□ 
Vol. Aide 

□□pn 

Other 

□ □□□ 



.Director 



: Program Name 



Location 



2.1 



Whole Class 



Teacher 

□□□□ 

Specialist 

□□□□ 

Paid Aide 

□ □□□ 
Vol, Aide 

□□□□ 

Other 

□□□□ 



Inst. 


Sup. 


7. 


% 


7. 


• 7. 


7. 


7. 


7. 


7. 


7. 


7. 



> 


3^1 Whole Class 

Inst. Sup. 


Director 


Teacher 7. 

□□□□ ~ 


7. 




% 




Specialist . 7. 


Program Name 


□□□□ ~ 

Paid Aide 7. 


7. 




□ □□□ ~ 


7. 


Location 


Vol. Aide 7. 
□ □□□ 




Other 7. 


7. 




□□□□ 












^•1 Whole Class 

Inst. Sup, 


Director 


Teacher % 


% 




gnna ~ 

Specialist 7, 

□ □□□ "~ 
Paid Aide 7. 




Program Name 


■7. 
7. 




□ □□□ ~. 
Vol, Aide 7. 

□ □□□ ~ 
Other 7. 




Location \ 


7. 




• 7. 


ERIC 3C2 







2*2 Smali Group 

Insto Sup. 



Teacher 

□□□□ 

Specialist 

□□□□ 

Paid Aide 

□ □□□ 
Vol. Aide 

Other 

□□QD 



_7. 
_7. 
_7, 

7. 



z3 



372 



Small Group 



Teacher 

□□□□ 

Specialist 

□□□□ 

paid Aide 

□ □□□ 
Vol. Aide 

□ □□□ 

Other 

□□□□ 



Inst. 


Sup. 


, 7. 


7. 


7, 


7. 


_% 


7. 


% 


7. 


7. 


7. 



4,2 



Small Group 

Inst, Sup, 



Teacher 

□□□□ 

Specialist 



specialist 

□ □□□ "~ 

Paid Aide 7. 

□ □□□ ~" 
Vol. Aide 7. 

□ □□□ 

Other 7. 

□ □□□ 
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7. 



,7. 

7« 



2.3 



Individual 



Teacher 

EIQDD 

Specialist 

□□□□ 

Paid Aide 

□□□□ 

Vol. Aide 

□ □□□ 
Other 

□□□□ 



Inst. 


Sup J 


7. 


i 


7. 




% 




TIL 


7 


7. 





trrr 



Specialist ' % 

□□□□ "~ 

PaifiLAide„ % 



Individual 

Inst._ Sup. 



Vol. Aide 

□ □□□v 
Other 

□□□□ 



.7. 
7. 



7 



,7. 
7. 



4.3 Individual 

Inat. Sup. 



ffifin 

Specialist - 7. 

□□□□ 

[Paid Aide 7. 

□ □□□ 

Vol. Aide 7. 

□ □□□ ~" 
Other 7. 

□□□□ 



